Greetings, I am having difficulty with exporting certain large files in Mathematica due to "Out of Memory" errors.
I am working with Digital Elevation Model (DEM) files, and the files need to be output as text in a defined format from Mathematica to be subsequently parsed by several feature analysis applications. If I was able to read and write a single file it would be sufficient, though ideally I would prefer to load 6-8 simultaneously with sizes ranging from 15MB to near 100MB.
I am working on a 3Ghz P4 with 4GB of RAM, and it doesn't seem that the application ends up using all 4GB of the memory when running - it seems to throw the Out of Memory error when it uses about 1.5 GB of memory (according to the XP task manager).
I have attempted using WriteString and Export to output the real-valued DEM matrix, with similar results with both (though Export is much faster to run out of memory). I am using MemoryConserve, the HistoryLength is set to 0 and I am using packed arrays as often as possible.
So the questions I have include:
1) Is there a defined limit to the amount of memory Mathematica uses before erroring out? And if so, can it be changed?
2) Other than the methods I'm already using, are there any other ways to squeeze extra efficient memory usage out of Mathematica, particularly during file I/O? Since I have to use the DEM format, DumpSave and Binary files are unfortunately not options.
3) Why exactly does it take over 1GB of Memory to output a 70MB text file after fairly minimal processing?
At this point, I've already re-written a decent chunk of the processing code in C, so I don't *have* to rely entirely on Mathematica, but it would be preferable to keep the majority of the work in Mathematica.
In any case, thank you in advance for any assistance,