Re: memory question
- To: mathgroup at smc.vnet.net
- Subject: [mg91251] Re: memory question
- From: David Bailey <dave at Remove_Thisdbailey.co.uk>
- Date: Mon, 11 Aug 2008 06:07:56 -0400 (EDT)
- References: <firstname.lastname@example.org>
Jan Schmedes wrote:
> Dear group,
> I am running Mathematica 6.0.3 on a Mac Pro with 4 Gig memory. I have
> fortran binary files that I can read in correctly (numbers of order
> 10^6) using Import[...,"Real32"]. But for larger problems, the size
> of the binary file is about 80 Mb, all memory is used (if i hit top,
> VSIZE is 17G) and only 2% of the CPU are used, so it basically can
> not handle the file. Is there a way to decrease the precision at
> which the file is read in, for my purpose the precision does not have
> to be very high?
> Thank you
I am a little puzzled because 80 MB of real numbers corresponds to 10^7
numbers, or slightly less if the binary records contain any red tape.
This is not very big - I can easily generate vectors much larger than
this on my 32-bit Windows XP machine.
This makes me wonder if you have already nearly filled your memory with
some other Mathematica structure prior to executing the Import command.
If you try to store Real numbers at reduced precision, you will not save
memory. However, it is important to realise that arrays (i.e, lists) of
Real numbers can be stored in packed or unpacked format. Packed arrays
are more efficient for most purposes and consume much less memory, but
in other respects they behave the same way. Try applying the function
Developer`PackedArrayQ to a smaller array read in by Import to determine
if your arrays are packed. I have noticed that Import does not always
pack its output, even when it should be able to do so.
Since your problem is probably related to other stuff that you have
created prior to using the Import command, I would check if you have any
other large unpacked arrays.
As others have said, if you don't need all that data in memory at once,
it may be better to read it a bit at a time.
Prev by Date:
Re: Help with StyleSheets
Next by Date:
Re: Simulating a Mouseover event in EventHandler
Previous by thread:
Re: memory question
Next by thread: