MathGroup Archive 2004

[Date Index] [Thread Index] [Author Index]

Search the Archive

Mathematica 5 I/O memory issues.

  • To: mathgroup at smc.vnet.net
  • Subject: [mg49312] Mathematica 5 I/O memory issues.
  • From: bryn at acm.augsburg.edu (B. Bue)
  • Date: Tue, 13 Jul 2004 04:32:47 -0400 (EDT)
  • Sender: owner-wri-mathgroup at wolfram.com

Greetings, I am having difficulty with exporting certain large files
in Mathematica due to "Out of Memory" errors.

I am working with Digital Elevation Model (DEM) files, and the files
need to be output as text in a defined format from Mathematica to be
subsequently parsed by several feature analysis applications.  If I
was able to read and write a single file it would be sufficient,
though ideally I would prefer to load 6-8 simultaneously with sizes
ranging from 15MB to near 100MB.

I am working on a 3Ghz P4 with 4GB of RAM, and it doesn't seem that
the application ends up using all 4GB of the memory when running - it
seems to throw the Out of Memory error when it uses about 1.5 GB of
memory (according to the XP task manager).

I have attempted using WriteString and Export to output the
real-valued DEM matrix, with similar results with both (though Export
is much faster to run out of memory). I am using MemoryConserve, the
HistoryLength is set to 0 and I am using packed arrays as often as
possible.

So the questions I have include: 
1) Is there a defined limit to the amount of memory Mathematica uses
before erroring out? And if so, can it be changed?

2) Other than the methods I'm already using, are there any other ways
to squeeze extra efficient memory usage out of Mathematica,
particularly during file I/O? Since I have to use the DEM format,
DumpSave and Binary files are unfortunately not options.

3) Why exactly does it take over 1GB of Memory to output a 70MB text
file after fairly minimal processing?

At this point, I've already re-written a decent chunk of the
processing code in C, so I don't *have* to rely entirely on
Mathematica, but it would be preferable to keep the majority of the
work in Mathematica.

In any case, thank you in advance for any assistance,
-Brian


  • Prev by Date: Re: ArcCos[x] with x > 1
  • Next by Date: RE : Sum of list elements
  • Previous by thread: Re: Solving the variable system of equations
  • Next by thread: Re: Mathematica 5 I/O memory issues.