Re: Working with huge text files with mathematica
- To: mathgroup at smc.vnet.net
- Subject: [mg56436] Re: Working with huge text files with mathematica
- From: Maxim <ab_def at prontomail.com>
- Date: Tue, 26 Apr 2005 01:32:38 -0400 (EDT)
- References: <d4clv2$357$1@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
On Sat, 23 Apr 2005 05:20:34 +0000 (UTC), Andrey Shevchuk <AShevchuk at Maplebank.com> wrote: > > Hi Everybody, > > I face a problem when try to read in a huge data file with my > Mathematica5.1 > > The file is approx. 4Gb large and was created by another Mathematica > application (So, I think Mathematica should be able to handle it). > > Now, if I try to use Read on a stream from this file and it returns > EndOfFile and nothing else! > I checked the file with the HexEdit (an editor for huge files) and it is > not > corrupt and has actually the data I need. > > A similar file (from the same application) but 1Gb works perfectly. > Any ideas? > > Does Mathematica have an internal limit for the file size (and if yes, > can > one somehow override this option) ? > > I would appreciate any feedback! > > > Yes, Mathematica 5.1.0 handles files larger than 2Gb incorrectly (at least under Windows XP). Read and BinaryRead always return EndOfFile and SetStreamPosition[stream, pos] with pos!=0 just generates SetStreamPosition::stmrng warning. At the same time, it is possible to write more than 2Gb of data to a file opened with OpenWrite or OpenAppend...which will make the file unreadable. Storing the data in several smaller files seems to be the only possible workaround. Maxim Rytin m.r at inbox.ru