[Date Index] [Thread Index] [Author Index]

Re: memory use

• To: mathgroup at smc.vnet.net
• Subject: [mg4877] Re: memory use
• From: Mark Evans <evans at gte.net>
• Date: Thu, 26 Sep 1996 22:42:29 -0400
• Organization: None
• Sender: owner-wri-mathgroup at wolfram.com

Russell Towle wrote:
>  
> I use the syntax, (biglist >> "filename"), but to my surprise, the kernel
> requires tons of memory to execute this ...
>

As <hberndt9 at mail.idt.net> indicated, Mathematica is converting your integer 
array to a string array, and that takes a lot of memory.

He suggested using a binary file format, and I have previously supplied you 
with code to read and write such files.

Another workaround might be to manually define the procedure Mathematica 
should follow for writing the numbers to disk.  You would open a stream and 
loop through the array; each iteration would convert one number to an 
equivalent string and write it to disk.  This technique would probably be very 
slow, but at least it would not eat up your whole memory partition.

I suggest not writing each string separately to disk (one file call per 
number), but instead letting the strings pile up in a queue until some 
reasonable queue length is attained, at which point you would write the whole 
queue to disk.  This technique could easily yield an order of magnitude speed 
improvement over individual writes, but it would put you in total control of 
memory consumption.  At the far extreme, you could even wait until the whole 
array has been converted, duplicating the behavior you observe now.  Then you 
would be sacrificing memory for speed, as the kernel does now.  This tradeoff 
is a classic computer-science phenomenon.  What I have proposed is to 
sacrifice a little speed for the sake of memory.  By writing small pieces at a 
time, you don't have to deal with one enormous chunk of data, but it takes a 
little longer to get everything onto disk.

Look forward to version 3.0, which will have a kernel memory dump command, 
DumpSave[].  This command will save expressions to disk in the internal format 
used by the kernel.  Be aware that the dump file format will vary across 
platforms.

Mark Evans
evans at gte.net


==== [MESSAGE SEPARATOR] ====