MathGroup Archive 2008

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: memory question

  • To: mathgroup at smc.vnet.net
  • Subject: [mg91232] Re: memory question
  • From: Jean-Marc Gulliet <jeanmarc.gulliet at gmail.com>
  • Date: Sun, 10 Aug 2008 01:54:03 -0400 (EDT)
  • Organization: The Open University, Milton Keynes, UK
  • References: <g7k088$i67$1@smc.vnet.net>

Jan Schmedes wrote:

> I am running Mathematica 6.0.3 on a Mac Pro with 4 Gig memory. I have  
> fortran binary files that I can read in correctly (numbers of order  
> 10^6) using Import[...,"Real32"]. But for larger problems, the size  
> of the binary file is about 80 Mb, all memory is used (if i hit top,  
> VSIZE is 17G)  and only 2% of the CPU are used, so it basically can  
> not handle the file. Is there a way to decrease the precision at  
> which the file is read in, for my purpose the precision does not have  
> to be very high?

The type "Real32" means that you are already using the smallest
representation for real numbers: single-precision hardware
floating-point numbers (4 bytes).

Now, that Mathematica can read and write single-precision floating-point 
numbers, which occupy only four bytes per number in a file located on 
the hard drive, does not mean that, having imported such a file in 
memory, Mathematica stores these numbers in memory as 4-byte numbers.

As far as I am aware of, Mathematica handles internally only 
double-precision hardware floating-point numbers, which occupy eight 
bytes on a 64-bit system.

Having said that, I have successfully imported (with the Import command) 
a binary file made of 30,000 records of 8,000 single-precision reals. 
Though the file takes 915.5 MB on the disk, the imported data occupy 
twice as much RAM. (See code below.)

Note that, since the file is made of only one type of number, 
Mathematica stores the data as packed array (data structure that 
requires less memory that the regular list).

So, either you large file is corrupted in some way, or it may have some 
value that are not/cannot be interpreted as single-precision 
floating-point number, thus dramatically increasing the memory needed to 
store a regular list. (Well, it is hard to tell without having the 
actual file at hand.)

I would suggest you try the code below on your system and report what 
are the results.

Also, you could try using BinaryRead rather that Import to gain a finer 
control on memory consumption by reading the file, possibly only 
partially, record by record.

In[1]:= $HistoryLength = 0;

In[2]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB

Out[2]= {9.44836 MB, 9.57955 MB}

In[3]:= fo = "file.dat";

In[4]:= Do[
  BinaryWrite[fo, RandomReal[{10^5, 10^7}, {8000}],
   "Real32"], {30000}]

In[5]:= Close[fo]

Out[5]= "file.dat"

In[6]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB

Out[6]= {12.49 MB, 12.68 MB}

In[7]:= FileByteCount["file.dat"]/2.^20 MB

Out[7]= 915.527 MB

In[8]:= data = Import["file.dat", "Real32"];

In[9]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB

Out[9]= {1843.72 MB, 1843.87 MB}

In[10]:= (data // ByteCount)/2.^20 MB

Out[10]= 1831.05 MB

Hope this helps,
-- Jean-Marc



  • Prev by Date: Re: Re: sum question
  • Next by Date: Re: Piecewise function involving matrices
  • Previous by thread: Re: memory question
  • Next by thread: Re: memory question