Re: out of memory reading large(?) file (Q:)

*To*: mathgroup at smc.vnet.net*Subject*: [mg18841] Re: out of memory reading large(?) file (Q:)*From*: sidles at u.washington.edu (John A. Sidles)*Date*: Thu, 22 Jul 1999 22:57:43 -0400*Organization*: University of Washington, Seattle*References*: <7lumlt$lf5@smc.vnet.net> <7m14aj$pfk@smc.vnet.net>*Sender*: owner-wri-mathgroup at wolfram.com

In article <7m14aj$pfk at smc.vnet.net>, P.J. Hinton <paulh at wolfram.com> wrote: >In article <7lumlt$lf5 at smc.vnet.net>, "iMic" <schaferk at communique.net> writes: > >> i am trying to read in a large file (6.3MB) of zeroes (0.0) and ones (1.0). >> after giving the kernel 50MB, the kernel reports out of memory. surely this >> amount of memory should be enough. the version is Mathematica 3.0 >> on a PowerMac 6500/225 with 96MB RAM. >> i have tried before (on much smaller files), using unformatted binary output >> of the file along with ReadBinary, only to find unbearable long read times >> (> 5Mins). > >How are the zeroes and ones stored in the data file? Are they stored >as bits, or does each byte in the file correspond to a single zero/one? >When you read the data from the file, do you store each entry as an >Integer, Real, or String? Is it really necessary to store all >of the data in the file in memory at once? > >-- >P.J. Hinton >Mathematica Programming Group paulh at wolfram.com >Wolfram Research, Inc. >Disclaimer: Opinions expressed herein are those of the author alone. There is a non-obvious but *very* fast and *very* memory efficient way to read a text file as a list of numbers -- it is an order of magnitude faster than "ReadList", and handles more general file formats to boot. This idiom is well-known to most Mathematica cogniscenti, and is rediscovered every few months by frustrated users --- so I guess it's time for us users to teach it (again) to you guys at Wolfram Inc headquarters! (* Open the file *) stream = OpenRead["fileName here"]; (* Read it in as one long string *) theDataString = ReadList[stream,Record,RecordSeparators->{}][[1]]; (* Convert it to a Mathematica expression *) theDataArray = ( "{{"<>StringReplace[ StringDrop[theDataString,-1], {"e"->"*10^", (* trick for reading exponential notation! *) "\t"->",", "\n"->"},\n{" }] <> "}}") //ToExpression; (* All done! Release the string *) Clear[theDataString]; In comparison to the above, you'll find ReadList[] to be slow, buggy, and a memory hog, to the point that (as the original poster "iMic", and I, and many other users have found) ReadList[] is simply unusable for importing large text files of data. My experience is that customer support invariably suggests using ReadBinary[], which is not an acceptable response, since many applications export only text files. So, I hereby challenge Wolfram Customer Support to come up with any workaround even half as fast as the above idiom for importing big text files. And then put the best idiom you've got up on your web site! By the way, PJ, how come these oft-encountered problems with ReadList[], and their workarounds, are not retained in Wolfram's collective memory from year-to-year? Please point out to your managers that this is a good example of a customer problem that occurs over-and-over again, and gets discussed by the user group each time, but somehow evades solution at the corporate level. Just part of my persistent -- and I hope stimulating to Wolfram management --- campaign for better Mathematica documentation. The above would make a good internal case study at Wolfram Research, as an example of how persistent customer problems with Mathematica have been --- at least in the past --- handled poorly or not-at-all. But hey, do I detect signs that the on-line technical FAQ is being expanded?! Let us hope! Best wishes ... John Sidles