Re: Working with huge text files with mathematica
- To: mathgroup at smc.vnet.net
- Subject: [mg56453] Re: [mg56378] Working with huge text files with mathematica
- From: Andrey Shevchuk <AShevchuk at Maplebank.com>
- Date: Tue, 26 Apr 2005 01:32:54 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
the problem seems to be a general one. I tried to read several other big text files, but Mathematica failed. it always returns EndOfFile! By the way, even a short program in VBA can do that. regards > -----Original Message----- > From: Andrey Shevchuk To: mathgroup at smc.vnet.net > Sent: Monday, April 25, 2005 9:43 AM > Subject: RE: [mg56378] Working with huge text files with mathematica > > And what is very interesting: > > A SetStreamPosition[stream, 1] tells me that the file does not have that > many characters !!! > But I can see them in the HexEdit!! > > -----Original Message----- > From: Andrey Shevchuk To: mathgroup at smc.vnet.net > Sent: Monday, April 25, 2005 9:28 AM > Subject: RE: [mg56378] Working with huge text files with > mathematica > > Hi, > > I have a double processor machine, a quite powerful one, but: > the same error occurs on every other machine we tried out here. > > And what is even more confusing : > I try to read the data by small pieces as you say ( using Read (..., > "String") or Read (..., "Byte") etc. from the stream ), actually the same > way I created the file ( but for creating I used Write(...)). > > The file is a product of one simulation and the problem appears at > approx. 2 Gb. If the simulation is run shorter the problem doesn't appear. > > Memory usage is minimal - around 15 Mb for the whole kernel, so it > should not be the problem, it just returns EndOfFile to the Read command. > > I seems extremely illogical to me, since the file was created in > Mathematica, then Mathematica should be able to read it, shouldn't it? > > any ideas? > > -----Original Message----- > From: Barthelet, Luc [SMTP:lucb at ea.com] To: mathgroup at smc.vnet.net > Sent: Sunday, April 24, 2005 8:39 PM > To: Andrey Shevchuk > Subject: RE: [mg56378] Working with huge text files > with mathematica > > Are you on a 64bit machine? > Otherwise at 4GB you do not have enough memory even with > virtual memory > to read the file. It was created because it was written by > small > segments. > Make sure you only try to read it small chunks at a time. > > -----Original Message----- > From: Andrey Shevchuk [mailto:AShevchuk at Maplebank.com] To: mathgroup at smc.vnet.net > Sent: Saturday, April 23, 2005 7:16 AM > Subject: [mg56453] [mg56378] Working with huge text files with > mathematica > > > Hi Everybody, > > I face a problem when try to read in a huge data file with > my > Mathematica5.1 > > The file is approx. 4Gb large and was created by another > Mathematica > application (So, I think Mathematica should be able to > handle it). > > Now, if I try to use Read on a stream from this file and it > returns > EndOfFile and nothing else! > I checked the file with the HexEdit (an editor for huge > files) and it is > not > corrupt and has actually the data I need. > > A similar file (from the same application) but 1Gb works > perfectly. > Any ideas? > > Does Mathematica have an internal limit for the file size > (and if yes, > can > one somehow override this option) ? > > I would appreciate any feedback! >