Re: Mathematica 5 I/O memory issues.
- To: mathgroup at smc.vnet.net
- Subject: [mg49319] Re: Mathematica 5 I/O memory issues.
- From: Jens-Peer Kuska <kuska at informatik.uni-leipzig.de>
- Date: Wed, 14 Jul 2004 07:29:24 -0400 (EDT)
- Organization: Universitaet Leipzig
- References: <email@example.com>
- Reply-to: kuska at informatik.uni-leipzig.de
- Sender: owner-wri-mathgroup at wolfram.com
Hi, 1) your 32 bit operating system limits the memory for the application you should switch to a 64 bit operating system and a 64 bit Mathematica 2) for the most Export command Mathematica call external programs via MathLink, this mean the Mathematica kernel and the export program have to store the data and you have at least two copys of the data in memory. It depends how the MathLink program work with the data and it is easy to get a third copy of the data in the MathLink buffer,in the application and in the Mathematica kernel 3) since the Export function has a similar function ExportString I would expect, the the MathLink binary construct a string with the textual print out of the data and write the final string on your hard drive. This woulk require mutch mor memory than a sequential print direct onto a file, but only the WRI programmer know what the MathLink binary for the export does exactly Regards Jens "B. Bue" wrote: > > Greetings, I am having difficulty with exporting certain large files > in Mathematica due to "Out of Memory" errors. > > I am working with Digital Elevation Model (DEM) files, and the files > need to be output as text in a defined format from Mathematica to be > subsequently parsed by several feature analysis applications. If I > was able to read and write a single file it would be sufficient, > though ideally I would prefer to load 6-8 simultaneously with sizes > ranging from 15MB to near 100MB. > > I am working on a 3Ghz P4 with 4GB of RAM, and it doesn't seem that > the application ends up using all 4GB of the memory when running - it > seems to throw the Out of Memory error when it uses about 1.5 GB of > memory (according to the XP task manager). > > I have attempted using WriteString and Export to output the > real-valued DEM matrix, with similar results with both (though Export > is much faster to run out of memory). I am using MemoryConserve, the > HistoryLength is set to 0 and I am using packed arrays as often as > possible. > > So the questions I have include: > 1) Is there a defined limit to the amount of memory Mathematica uses > before erroring out? And if so, can it be changed? > > 2) Other than the methods I'm already using, are there any other ways > to squeeze extra efficient memory usage out of Mathematica, > particularly during file I/O? Since I have to use the DEM format, > DumpSave and Binary files are unfortunately not options. > > 3) Why exactly does it take over 1GB of Memory to output a 70MB text > file after fairly minimal processing? > > At this point, I've already re-written a decent chunk of the > processing code in C, so I don't *have* to rely entirely on > Mathematica, but it would be preferable to keep the majority of the > work in Mathematica. > > In any case, thank you in advance for any assistance, > -Brian