Re: memory use
- To: mathgroup at smc.vnet.net
- Subject: [mg4877] Re: memory use
- From: Mark Evans <evans at gte.net>
- Date: Thu, 26 Sep 1996 22:42:29 -0400
- Organization: None
- Sender: owner-wri-mathgroup at wolfram.com
Russell Towle wrote: > > I use the syntax, (biglist >> "filename"), but to my surprise, the kernel > requires tons of memory to execute this ... > As <hberndt9 at mail.idt.net> indicated, Mathematica is converting your integer array to a string array, and that takes a lot of memory. He suggested using a binary file format, and I have previously supplied you with code to read and write such files. Another workaround might be to manually define the procedure Mathematica should follow for writing the numbers to disk. You would open a stream and loop through the array; each iteration would convert one number to an equivalent string and write it to disk. This technique would probably be very slow, but at least it would not eat up your whole memory partition. I suggest not writing each string separately to disk (one file call per number), but instead letting the strings pile up in a queue until some reasonable queue length is attained, at which point you would write the whole queue to disk. This technique could easily yield an order of magnitude speed improvement over individual writes, but it would put you in total control of memory consumption. At the far extreme, you could even wait until the whole array has been converted, duplicating the behavior you observe now. Then you would be sacrificing memory for speed, as the kernel does now. This tradeoff is a classic computer-science phenomenon. What I have proposed is to sacrifice a little speed for the sake of memory. By writing small pieces at a time, you don't have to deal with one enormous chunk of data, but it takes a little longer to get everything onto disk. Look forward to version 3.0, which will have a kernel memory dump command, DumpSave[]. This command will save expressions to disk in the internal format used by the kernel. Be aware that the dump file format will vary across platforms. Mark Evans evans at gte.net ==== [MESSAGE SEPARATOR] ====