Re: Mathematica 5 I/O memory issues.
- To: mathgroup at smc.vnet.net
- Subject: [mg49336] Re: [mg49312] Mathematica 5 I/O memory issues.
- From: Yasvir Tesiram <yat at omrf.ouhsc.edu>
- Date: Thu, 15 Jul 2004 06:59:58 -0400 (EDT)
- References: <200407130832.EAA09640@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
Hi, One would think that these are small files in comparison to the amount of RAM. But Mathematica has to hold it all somewhere and then do something with it when you want to process it. Most of this such as Partition is pretty efficient. See what MemoryInUse returns before and after such routines. Displaying Graphics is the killer. When working with large files, avoid it at all costs or set DisplayFunction -> Identity. Instead of Export, you could probably use OpenWrite and Write. You could work with smaller chunks of the file. Use SetStreamPosition to start at a particular point. I guess you have basically gone through most of what Mathematica offers in terms of I/O. I tried I/O with ReadList etc on a 100 MB file to check and was able to essentially, Read, Partition, Display as a DensityPlot and write it out as a text file. Any more and I would run out of Memory according to Mathematica. Cheers Yas On Jul 13, 2004, at 3:32 AM, B. Bue wrote: > Greetings, I am having difficulty with exporting certain large files > in Mathematica due to "Out of Memory" errors. > > I am working with Digital Elevation Model (DEM) files, and the files > need to be output as text in a defined format from Mathematica to be > subsequently parsed by several feature analysis applications. If I > was able to read and write a single file it would be sufficient, > though ideally I would prefer to load 6-8 simultaneously with sizes > ranging from 15MB to near 100MB. > > I am working on a 3Ghz P4 with 4GB of RAM, and it doesn't seem that > the application ends up using all 4GB of the memory when running - it > seems to throw the Out of Memory error when it uses about 1.5 GB of > memory (according to the XP task manager). > > I have attempted using WriteString and Export to output the > real-valued DEM matrix, with similar results with both (though Export > is much faster to run out of memory). I am using MemoryConserve, the > HistoryLength is set to 0 and I am using packed arrays as often as > possible. > > So the questions I have include: > 1) Is there a defined limit to the amount of memory Mathematica uses > before erroring out? And if so, can it be changed? > > 2) Other than the methods I'm already using, are there any other ways > to squeeze extra efficient memory usage out of Mathematica, > particularly during file I/O? Since I have to use the DEM format, > DumpSave and Binary files are unfortunately not options. > > 3) Why exactly does it take over 1GB of Memory to output a 70MB text > file after fairly minimal processing? > > At this point, I've already re-written a decent chunk of the > processing code in C, so I don't *have* to rely entirely on > Mathematica, but it would be preferable to keep the majority of the > work in Mathematica. > > In any case, thank you in advance for any assistance, > -Brian
- References:
- Mathematica 5 I/O memory issues.
- From: bryn@acm.augsburg.edu (B. Bue)
- Mathematica 5 I/O memory issues.