Re: memory question
- To: mathgroup at smc.vnet.net
- Subject: [mg91232] Re: memory question
- From: Jean-Marc Gulliet <jeanmarc.gulliet at gmail.com>
- Date: Sun, 10 Aug 2008 01:54:03 -0400 (EDT)
- Organization: The Open University, Milton Keynes, UK
- References: <g7k088$i67$1@smc.vnet.net>
Jan Schmedes wrote:
> I am running Mathematica 6.0.3 on a Mac Pro with 4 Gig memory. I have
> fortran binary files that I can read in correctly (numbers of order
> 10^6) using Import[...,"Real32"]. But for larger problems, the size
> of the binary file is about 80 Mb, all memory is used (if i hit top,
> VSIZE is 17G) and only 2% of the CPU are used, so it basically can
> not handle the file. Is there a way to decrease the precision at
> which the file is read in, for my purpose the precision does not have
> to be very high?
The type "Real32" means that you are already using the smallest
representation for real numbers: single-precision hardware
floating-point numbers (4 bytes).
Now, that Mathematica can read and write single-precision floating-point
numbers, which occupy only four bytes per number in a file located on
the hard drive, does not mean that, having imported such a file in
memory, Mathematica stores these numbers in memory as 4-byte numbers.
As far as I am aware of, Mathematica handles internally only
double-precision hardware floating-point numbers, which occupy eight
bytes on a 64-bit system.
Having said that, I have successfully imported (with the Import command)
a binary file made of 30,000 records of 8,000 single-precision reals.
Though the file takes 915.5 MB on the disk, the imported data occupy
twice as much RAM. (See code below.)
Note that, since the file is made of only one type of number,
Mathematica stores the data as packed array (data structure that
requires less memory that the regular list).
So, either you large file is corrupted in some way, or it may have some
value that are not/cannot be interpreted as single-precision
floating-point number, thus dramatically increasing the memory needed to
store a regular list. (Well, it is hard to tell without having the
actual file at hand.)
I would suggest you try the code below on your system and report what
are the results.
Also, you could try using BinaryRead rather that Import to gain a finer
control on memory consumption by reading the file, possibly only
partially, record by record.
In[1]:= $HistoryLength = 0;
In[2]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB
Out[2]= {9.44836 MB, 9.57955 MB}
In[3]:= fo = "file.dat";
In[4]:= Do[
BinaryWrite[fo, RandomReal[{10^5, 10^7}, {8000}],
"Real32"], {30000}]
In[5]:= Close[fo]
Out[5]= "file.dat"
In[6]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB
Out[6]= {12.49 MB, 12.68 MB}
In[7]:= FileByteCount["file.dat"]/2.^20 MB
Out[7]= 915.527 MB
In[8]:= data = Import["file.dat", "Real32"];
In[9]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB
Out[9]= {1843.72 MB, 1843.87 MB}
In[10]:= (data // ByteCount)/2.^20 MB
Out[10]= 1831.05 MB
Hope this helps,
-- Jean-Marc