MathGroup Archive 2008

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: memory question

  • To: mathgroup at smc.vnet.net
  • Subject: [mg91251] Re: memory question
  • From: David Bailey <dave at Remove_Thisdbailey.co.uk>
  • Date: Mon, 11 Aug 2008 06:07:56 -0400 (EDT)
  • References: <g7k088$i67$1@smc.vnet.net>

Jan Schmedes wrote:
> Dear group,
> 
> I am running Mathematica 6.0.3 on a Mac Pro with 4 Gig memory. I have  
> fortran binary files that I can read in correctly (numbers of order  
> 10^6) using Import[...,"Real32"]. But for larger problems, the size  
> of the binary file is about 80 Mb, all memory is used (if i hit top,  
> VSIZE is 17G)  and only 2% of the CPU are used, so it basically can  
> not handle the file. Is there a way to decrease the precision at  
> which the file is read in, for my purpose the precision does not have  
> to be very high?
> 
> Thank you
> 
> Jan
> 

I am a little puzzled because 80 MB of real numbers corresponds to 10^7 
numbers, or slightly less if the binary records contain any red tape. 
This is not very big - I can easily generate vectors much larger than 
this on my 32-bit Windows XP machine.

This makes me wonder if you have already nearly filled your memory with 
some other Mathematica structure prior to executing the Import command.

If you try to store Real numbers at reduced precision, you will not save 
memory. However, it is important to realise that arrays (i.e, lists) of 
Real numbers can be stored in packed or unpacked format. Packed arrays 
are more efficient for most purposes and consume much less memory, but 
in other respects they behave the same way. Try applying the function 
Developer`PackedArrayQ to a smaller array read in by Import to determine 
if your arrays are packed. I have noticed that Import does not always 
pack its output, even when it should be able to do so.

Since your problem is probably related to other stuff that you have 
created prior to using the Import command, I would check if you have any 
other large unpacked arrays.

As others have said, if you don't need all that data in memory at once, 
it may be better to read it a bit at a time.

David Bailey
http://www.dbaileyconsultancy.co.uk


  • Prev by Date: Re: Help with StyleSheets
  • Next by Date: Re: Simulating a Mouseover event in EventHandler
  • Previous by thread: Re: memory question
  • Next by thread: correction