Re: Re: out of memory reading large(?) file (Q:)

*To*: mathgroup at smc.vnet.net*Subject*: [mg19358] Re: [mg18841] Re: out of memory reading large(?) file (Q:)*From*: David Withoff <withoff at wolfram.com>*Date*: Fri, 20 Aug 1999 23:09:36 -0400*Sender*: owner-wri-mathgroup at wolfram.com

I finally got time to look at a message that was posted to mathgroup about a month ago (22 Jul 1999) and which included some claims and suggestions about ReadList that are not generally correct: John A. Sidles <sidles at u.washington.edu> wrote: > There is a non-obvious but *very* fast and *very* memory > efficient way to read a text file as a list of numbers -- it > is an order of magnitude faster than "ReadList", and handles > more general file formats to boot. This idiom is well-known to > most Mathematica cogniscenti, and is rediscovered every few months > by frustrated users --- so I guess it's time for us users to > teach it (again) to you guys at Wolfram Inc headquarters! > > (* Open the file *) > stream = OpenRead["fileName here"]; > > (* Read it in as one long string *) > theDataString = ReadList[stream,Record,RecordSeparators->{}][[1]]; > > (* Convert it to a Mathematica expression *) > theDataArray = ( > "{{"<>StringReplace[ > StringDrop[theDataString,-1], > {"e"->"*10^", (* trick for reading exponential notation! *) > "\t"->",", > "\n"->"},\n{" > }] <> "}}") //ToExpression; > > (* All done! Release the string *) > Clear[theDataString]; > > In comparison to the above, you'll find ReadList[] to be > slow, buggy, and a memory hog, to the point that (as the > original poster "iMic", and I, and many other users have found) > ReadList[] is simply unusable for importing large text files > of data. In typical examples that would be covered by the suggested alternative approach (reading the file into one long string and converting that string to an expression using ToExpression), ReadList by itself is faster and uses less memory than this alternative. For example, since the message that led to this thread referred to a large file of zeroes and ones: >In article <7lumlt$lf5 at smc.vnet.net>, "iMic" <schaferk at communique.net> writes: >> i am trying to read in a large file (6.3MB) of zeroes (0.0) and ones (1.0). >> after giving the kernel 50MB, the kernel reports out of memory. surely this >> amount of memory should be enough. the version is Mathematica 3.0 >> on a PowerMac 6500/225 with 96MB RAM. >> i have tried before (on much smaller files), using unformatted binary output >> of the file along with ReadBinary, only to find unbearable long read times >> (> 5Mins). I did an experiment with a large file of zeroes and ones, which I generated using Do[Write["file", OutputForm[Infix[{1.0, 0.0, 1.0, 0.0}, "\t"]]], {100000}] I read this file into an array using ReadList, in the current version of Mathematica (Version 4) for Linux: In[1]:= m = {MemoryInUse[], MaxMemoryUsed[]} ; In[2]:= Timing[result = ReadList["file", Number, RecordLists->True]][[1]] Out[2]= 5.07 Second In[3]:= {MemoryInUse[], MaxMemoryUsed[]} - m Out[3]= {10018392, 10334560} and for comparison read the same file into an equivalent matrix using the suggested alternative (reading the file into one long string and using ToExpression to convert that string into an expression): In[1]:= m = {MemoryInUse[], MaxMemoryUsed[]} ; In[2]:= Timing[ stream = OpenRead["file"]; theDataString = ReadList[stream,Record,RecordSeparators->{}][[1]]; Close[stream]; theDataArray = ( "{{"<>StringReplace[ StringDrop[theDataString,-1], {"e"->"*10^", (* trick for reading exponential notation! *) "\t"->",", "\n"->"},\n{" }] <> "}}") //ToExpression; Clear[theDataString]][[1]] Out[2]= 12.64 Second In[3]:= {MemoryInUse[], MaxMemoryUsed[]} - m Out[3]= {10020004, 14863856} In this example ReadList by itself is about twice as fast as the suggested alternative and uses less memory. The alternative uses about four megabytes of extra memory for intermediate results, while the extra memory (memory beyond that needed to store the final result) used by ReadList alone was negligible. I found similar behavior for all of the examples that I tried, both in Version 3 and in Version 4 of Mathematica, and on other types of computers (such as Mathematica for Windows). ReadList by itself is consistently faster than the suggested alternative and uses less memory -- sometimes a lot less, especially for numbers in "e" (Fortran) notation. (In this regard, if the suggested alternative is used, the replacement "e"->"*10^" really should be "e"->"*^".) Although reading the entire file into one long string and processing that string is not a recommended approach in general, that approach is useful in some other examples that do not involve duplicating the basic functionality of ReadList. There are also quite a number of other approaches that are of interest in other examples. It is always possible to read data in to Mathematica in a way that is within a tolerable factor of optimal. The people in the technical support group at Wolfram Research freqently can offer assistance with this sort of thing. Dave Withoff Wolfram Research

**Re: Control Function With NDsolve**

**Fontproblems**

**Re: Tricky Symbolizations with the Notation Package**

**Fontproblems**