Re: Question about memory consumption in my codes by using Mathematica 6, Thanks!
- To: mathgroup at smc.vnet.net
- Subject: [mg80973] Re: Question about memory consumption in my codes by using Mathematica 6, Thanks!
- From: Aranthon <a.dwarf at gmail.com>
- Date: Thu, 6 Sep 2007 05:37:42 -0400 (EDT)
- References: <f9s47u$ao7$1@smc.vnet.net>
It sounds like I'm doing very much the same thing, and I'm finding
that Mathematica is extremely sensitive to the details of how I carry
out the reading and analysis.
In my case, the data files are in the neighborhood of 7 million data
points (the raw binary files are up to 80 MB). The data consists
primarily of records in the following format: {Real32, Integer8,
Integer8, Integer16}. If I read the data straight in, it takes as
much as 10 minutes and invariably runs out of memory - on a 3+ GHz P4
with 2 GB of RAM. I think my big problem is that Mathematica doesn't
like the mixed number types - it would much prefer to have everything
the same data type. After a lot of tweaking, I can read in that same
datafile in about 30 s and without even remotely running out of
memory. I have to do some pretty funky things to make it work (I read
in the whole dataset as Integer8, break off the portion that ought to
be Real32, then write that to a temporary binary file and re-read it
as Real 32. I also rewrote the part that operates on the Integer16 to
operate on the resulting two Integer8s. From a comp-sci perspective,
I think it's kind of sketchy, but it works).
My main suggestions would be to try not to use multiple number formats
in the same list structure, and to try to avoid building up any
extraneous intermediate lists (e.g. use Scan rather than Map if
possible). Also, use ByteCount[], MemoryInUse[], and MaxMemoryUsed[]
when you're prototyping your algorithm, rather than using the Windows
Task Manager, so you know exactly how much memory each portion of your
analysis consumes. Oh, and you can use MemoryConstrained[] to kill
your evaluation if the memory use is getting out of control so you
don't lose the rest of your session.
Good luck!
Greg
On Aug 14, 7:38 am, foxvexon <yulin... at gmail.com> wrote:
> I write a routine that repeats loading and treating a large amount of
> data imported from external data files produced by another program. I
> enclose codes for this task in a simple Module[{local_vars},
> expressions]. Of'cause, it calls pre-defined routines outside but does
> not introduce any new variable except temple local ones.
>
> I am confused that If this is done for just one or two loops/times, it
> works well. But if I hope to run this loop several times, like 10~200,
> Mathematica usually reports memory consumed out and terminates
> automatically.
>
> I check my codes over and over to make sure that I do not introduce
> new variables in each loop that will eat up memory every time. But, I
> do notice (from windows task manager) that the used memory by
> Mathematica 6 is increasing with each loop.
>
> Please help! Thanks a lot in advance!