Re: Working with huge text files with mathematica
- To: mathgroup at smc.vnet.net
- Subject: [mg56439] Re: [mg56378] Working with huge text files with mathematica
- From: Andrey Shevchuk <AShevchuk at Maplebank.com>
- Date: Tue, 26 Apr 2005 01:32:40 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
Hi, I have a double processor machine, a quite powerful one, but: the same error occurs on every other machine we tried out here. And what is even more confusing : I try to read the data by small pieces as you say ( using Read (..., "String") or Read (..., "Byte") etc. from the stream ), actually the same way I created the file ( but for creating I used Write(...)). The file is a product of one simulation and the problem appears at approx. 2 Gb. If the simulation is run shorter the problem doesn't appear. Memory usage is minimal - around 15 Mb for the whole kernel, so it should not be the problem, it just returns EndOfFile to the Read command. I seems extremely illogical to me, since the file was created in Mathematica, then Mathematica should be able to read it, shouldn't it? any ideas? > -----Original Message----- > From: Barthelet, Luc [SMTP:lucb at ea.com] To: mathgroup at smc.vnet.net > Sent: Sunday, April 24, 2005 8:39 PM > Subject: RE: [mg56378] Working with huge text files with mathematica > > Are you on a 64bit machine? > Otherwise at 4GB you do not have enough memory even with virtual memory > to read the file. It was created because it was written by small > segments. > Make sure you only try to read it small chunks at a time. > > -----Original Message----- > From: Andrey Shevchuk [mailto:AShevchuk at Maplebank.com] To: mathgroup at smc.vnet.net > Sent: Saturday, April 23, 2005 7:16 AM > Subject: [mg56439] [mg56378] Working with huge text files with mathematica > > > Hi Everybody, > > I face a problem when try to read in a huge data file with my > Mathematica5.1 > > The file is approx. 4Gb large and was created by another Mathematica > application (So, I think Mathematica should be able to handle it). > > Now, if I try to use Read on a stream from this file and it returns > EndOfFile and nothing else! > I checked the file with the HexEdit (an editor for huge files) and it is > not > corrupt and has actually the data I need. > > A similar file (from the same application) but 1Gb works perfectly. > Any ideas? > > Does Mathematica have an internal limit for the file size (and if yes, > can > one somehow override this option) ? > > I would appreciate any feedback! > >