[Date Index] [Thread Index] [Author Index]

Re: Working with huge text files with mathematica

• To: mathgroup at smc.vnet.net
• Subject: [mg56481] Re: Working with huge text files with mathematica
• From: David Bailey <dave at Remove_Thisdbailey.co.uk>
• Date: Tue, 26 Apr 2005 21:52:35 -0400 (EDT)
• References: <d4clv2$357$1@smc.vnet.net>
• Sender: owner-wri-mathgroup at wolfram.com

Andrey Shevchuk wrote:
> Hi Everybody,
> 
> I face a problem when try to read in a huge data file with my Mathematica5.1
> 
> The file is approx. 4Gb large and was created by another Mathematica
> application (So, I think Mathematica should be able to handle it). 
> 
> Now, if I try to use Read on a stream from this file and it returns
> EndOfFile and nothing else!
> I checked the file with the HexEdit (an editor for huge files) and it is not
> corrupt and has actually the data I need.
> 
> A similar file (from the same application) but 1Gb works perfectly.
> Any ideas?
> 
> Does Mathematica have an internal limit for the file size (and if yes, can
> one somehow override this option) ?
> 
> I would appreciate any feedback!
> 
> 
> 
It would seem from what others have written that there is a 2G limit on 
the size of files that can be read by Mathematica under Windows. This 
must be a bug in Mathematica - Windows itself can exceed this limit. 
Rather than changing operating systems or splitting the file, you might 
want to try using J/Link and reading the file via Java. For best 
performance, you might find it was better to read the file in 
substantial chunks and buffer it inside Mathematica, however.

David Bailey
dbaileyconsultancy.co.uk