[Date Index] [Thread Index] [Author Index]

Re: Working with huge text files with mathematica

• To: mathgroup at smc.vnet.net
• Subject: [mg56440] Re: [mg56378] Working with huge text files with mathematica
• From: Andrey Shevchuk <AShevchuk at Maplebank.com>
• Date: Tue, 26 Apr 2005 01:32:40 -0400 (EDT)
• Sender: owner-wri-mathgroup at wolfram.com

And what is very interesting:

A SetStreamPosition[stream, 1] tells me that the file does not have that
many characters !!! 
But I can see them in the HexEdit!!

> -----Original Message-----
> From:	Andrey Shevchuk 
To: mathgroup at smc.vnet.net
> Sent:	Monday, April 25, 2005 9:28 AM
> Subject:	RE: [mg56378] Working with huge text files with mathematica
> 
> Hi,
> 
> I have a double processor machine, a quite powerful one, but:
> the same error occurs on every other machine we tried out here.
> 
> And what is even more confusing :
> I try to read the data by small pieces as you say ( using Read (...,
> "String") or Read (..., "Byte") etc. from the stream ), actually the same
> way I created the file ( but for creating I used Write(...)).
> 
> The file is a product of one simulation and the problem appears at approx.
> 2 Gb. If the simulation is run shorter the problem doesn't appear.
> 
> Memory usage is minimal - around 15 Mb for the whole kernel, so it should
> not be the problem, it just returns EndOfFile to the Read command. 
> 
> I seems extremely illogical to me, since the file was created in
> Mathematica, then Mathematica should be able to read it, shouldn't it? 
> 
> any ideas?
> 
> 	-----Original Message-----
> 	From:	Barthelet, Luc [SMTP:lucb at ea.com]
To: mathgroup at smc.vnet.net
> 	Sent:	Sunday, April 24, 2005 8:39 PM
> 	To:	Andrey Shevchuk
> 	Subject:	RE: [mg56378] Working with huge text files with
> mathematica
> 
> 	Are you on a 64bit machine?
> 	Otherwise at 4GB you do not have enough memory even with virtual
> memory
> 	to read the file. It was created because it was written by small
> 	segments.
> 	Make sure you only try to read it small chunks at a time.
> 
> 	-----Original Message-----
> 	From: Andrey Shevchuk [mailto:AShevchuk at Maplebank.com] 
To: mathgroup at smc.vnet.net
> 	Sent: Saturday, April 23, 2005 7:16 AM
> 	Subject: [mg56440] [mg56378] Working with huge text files with mathematica
> 
> 
> 	Hi Everybody,
> 
> 	I face a problem when try to read in a huge data file with my
> 	Mathematica5.1
> 
> 	The file is approx. 4Gb large and was created by another Mathematica
> 	application (So, I think Mathematica should be able to handle it). 
> 
> 	Now, if I try to use Read on a stream from this file and it returns
> 	EndOfFile and nothing else!
> 	I checked the file with the HexEdit (an editor for huge files) and
> it is
> 	not
> 	corrupt and has actually the data I need.
> 
> 	A similar file (from the same application) but 1Gb works perfectly.
> 	Any ideas?
> 
> 	Does Mathematica have an internal limit for the file size (and if
> yes,
> 	can
> 	one somehow override this option) ?
> 
> 	I would appreciate any feedback!
> 
>