Re: memory question
- To: mathgroup at smc.vnet.net
- Subject: [mg91251] Re: memory question
- From: David Bailey <dave at Remove_Thisdbailey.co.uk>
- Date: Mon, 11 Aug 2008 06:07:56 -0400 (EDT)
- References: <g7k088$i67$1@smc.vnet.net>
Jan Schmedes wrote: > Dear group, > > I am running Mathematica 6.0.3 on a Mac Pro with 4 Gig memory. I have > fortran binary files that I can read in correctly (numbers of order > 10^6) using Import[...,"Real32"]. But for larger problems, the size > of the binary file is about 80 Mb, all memory is used (if i hit top, > VSIZE is 17G) and only 2% of the CPU are used, so it basically can > not handle the file. Is there a way to decrease the precision at > which the file is read in, for my purpose the precision does not have > to be very high? > > Thank you > > Jan > I am a little puzzled because 80 MB of real numbers corresponds to 10^7 numbers, or slightly less if the binary records contain any red tape. This is not very big - I can easily generate vectors much larger than this on my 32-bit Windows XP machine. This makes me wonder if you have already nearly filled your memory with some other Mathematica structure prior to executing the Import command. If you try to store Real numbers at reduced precision, you will not save memory. However, it is important to realise that arrays (i.e, lists) of Real numbers can be stored in packed or unpacked format. Packed arrays are more efficient for most purposes and consume much less memory, but in other respects they behave the same way. Try applying the function Developer`PackedArrayQ to a smaller array read in by Import to determine if your arrays are packed. I have noticed that Import does not always pack its output, even when it should be able to do so. Since your problem is probably related to other stuff that you have created prior to using the Import command, I would check if you have any other large unpacked arrays. As others have said, if you don't need all that data in memory at once, it may be better to read it a bit at a time. David Bailey http://www.dbaileyconsultancy.co.uk