Re: Working with huge text files with mathematica
- To: mathgroup at smc.vnet.net
- Subject: [mg56440] Re: [mg56378] Working with huge text files with mathematica
- From: Andrey Shevchuk <AShevchuk at Maplebank.com>
- Date: Tue, 26 Apr 2005 01:32:40 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
And what is very interesting: A SetStreamPosition[stream, 1] tells me that the file does not have that many characters !!! But I can see them in the HexEdit!! > -----Original Message----- > From: Andrey Shevchuk To: mathgroup at smc.vnet.net > Sent: Monday, April 25, 2005 9:28 AM > Subject: RE: [mg56378] Working with huge text files with mathematica > > Hi, > > I have a double processor machine, a quite powerful one, but: > the same error occurs on every other machine we tried out here. > > And what is even more confusing : > I try to read the data by small pieces as you say ( using Read (..., > "String") or Read (..., "Byte") etc. from the stream ), actually the same > way I created the file ( but for creating I used Write(...)). > > The file is a product of one simulation and the problem appears at approx. > 2 Gb. If the simulation is run shorter the problem doesn't appear. > > Memory usage is minimal - around 15 Mb for the whole kernel, so it should > not be the problem, it just returns EndOfFile to the Read command. > > I seems extremely illogical to me, since the file was created in > Mathematica, then Mathematica should be able to read it, shouldn't it? > > any ideas? > > -----Original Message----- > From: Barthelet, Luc [SMTP:lucb at ea.com] To: mathgroup at smc.vnet.net > Sent: Sunday, April 24, 2005 8:39 PM > To: Andrey Shevchuk > Subject: RE: [mg56378] Working with huge text files with > mathematica > > Are you on a 64bit machine? > Otherwise at 4GB you do not have enough memory even with virtual > memory > to read the file. It was created because it was written by small > segments. > Make sure you only try to read it small chunks at a time. > > -----Original Message----- > From: Andrey Shevchuk [mailto:AShevchuk at Maplebank.com] To: mathgroup at smc.vnet.net > Sent: Saturday, April 23, 2005 7:16 AM > Subject: [mg56440] [mg56378] Working with huge text files with mathematica > > > Hi Everybody, > > I face a problem when try to read in a huge data file with my > Mathematica5.1 > > The file is approx. 4Gb large and was created by another Mathematica > application (So, I think Mathematica should be able to handle it). > > Now, if I try to use Read on a stream from this file and it returns > EndOfFile and nothing else! > I checked the file with the HexEdit (an editor for huge files) and > it is > not > corrupt and has actually the data I need. > > A similar file (from the same application) but 1Gb works perfectly. > Any ideas? > > Does Mathematica have an internal limit for the file size (and if > yes, > can > one somehow override this option) ? > > I would appreciate any feedback! > >