Re: memory question

*To*: mathgroup at smc.vnet.net*Subject*: [mg91232] Re: memory question*From*: Jean-Marc Gulliet <jeanmarc.gulliet at gmail.com>*Date*: Sun, 10 Aug 2008 01:54:03 -0400 (EDT)*Organization*: The Open University, Milton Keynes, UK*References*: <g7k088$i67$1@smc.vnet.net>

Jan Schmedes wrote: > I am running Mathematica 6.0.3 on a Mac Pro with 4 Gig memory. I have > fortran binary files that I can read in correctly (numbers of order > 10^6) using Import[...,"Real32"]. But for larger problems, the size > of the binary file is about 80 Mb, all memory is used (if i hit top, > VSIZE is 17G) and only 2% of the CPU are used, so it basically can > not handle the file. Is there a way to decrease the precision at > which the file is read in, for my purpose the precision does not have > to be very high? The type "Real32" means that you are already using the smallest representation for real numbers: single-precision hardware floating-point numbers (4 bytes). Now, that Mathematica can read and write single-precision floating-point numbers, which occupy only four bytes per number in a file located on the hard drive, does not mean that, having imported such a file in memory, Mathematica stores these numbers in memory as 4-byte numbers. As far as I am aware of, Mathematica handles internally only double-precision hardware floating-point numbers, which occupy eight bytes on a 64-bit system. Having said that, I have successfully imported (with the Import command) a binary file made of 30,000 records of 8,000 single-precision reals. Though the file takes 915.5 MB on the disk, the imported data occupy twice as much RAM. (See code below.) Note that, since the file is made of only one type of number, Mathematica stores the data as packed array (data structure that requires less memory that the regular list). So, either you large file is corrupted in some way, or it may have some value that are not/cannot be interpreted as single-precision floating-point number, thus dramatically increasing the memory needed to store a regular list. (Well, it is hard to tell without having the actual file at hand.) I would suggest you try the code below on your system and report what are the results. Also, you could try using BinaryRead rather that Import to gain a finer control on memory consumption by reading the file, possibly only partially, record by record. In[1]:= $HistoryLength = 0; In[2]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB Out[2]= {9.44836 MB, 9.57955 MB} In[3]:= fo = "file.dat"; In[4]:= Do[ BinaryWrite[fo, RandomReal[{10^5, 10^7}, {8000}], "Real32"], {30000}] In[5]:= Close[fo] Out[5]= "file.dat" In[6]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB Out[6]= {12.49 MB, 12.68 MB} In[7]:= FileByteCount["file.dat"]/2.^20 MB Out[7]= 915.527 MB In[8]:= data = Import["file.dat", "Real32"]; In[9]:= {MemoryInUse[], MaxMemoryUsed[]}/2.^20 MB Out[9]= {1843.72 MB, 1843.87 MB} In[10]:= (data // ByteCount)/2.^20 MB Out[10]= 1831.05 MB Hope this helps, -- Jean-Marc