Services & Resources / Wolfram Forums
-----
 /
MathGroup Archive
2005
*January
*February
*March
*April
*May
*June
*July
*August
*September
*October
*November
*December
*Archive Index
*Ask about this page
*Print this page
*Give us feedback
*Sign up for the Wolfram Insider

MathGroup Archive 2005

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Working with huge text files with mathematica

  • To: mathgroup at smc.vnet.net
  • Subject: [mg56439] Re: [mg56378] Working with huge text files with mathematica
  • From: Andrey Shevchuk <AShevchuk at Maplebank.com>
  • Date: Tue, 26 Apr 2005 01:32:40 -0400 (EDT)
  • Sender: owner-wri-mathgroup at wolfram.com

Hi,

I have a double processor machine, a quite powerful one, but:
the same error occurs on every other machine we tried out here.

And what is even more confusing :
I try to read the data by small pieces as you say ( using Read (...,
"String") or Read (..., "Byte") etc. from the stream ), actually the same
way I created the file ( but for creating I used Write(...)).

The file is a product of one simulation and the problem appears at approx. 2
Gb. If the simulation is run shorter the problem doesn't appear.

Memory usage is minimal - around 15 Mb for the whole kernel, so it should
not be the problem, it just returns EndOfFile to the Read command. 

I seems extremely illogical to me, since the file was created in
Mathematica, then Mathematica should be able to read it, shouldn't it? 

any ideas?

> -----Original Message-----
> From:	Barthelet, Luc [SMTP:lucb at ea.com]
To: mathgroup at smc.vnet.net
> Sent:	Sunday, April 24, 2005 8:39 PM
> Subject:	RE: [mg56378] Working with huge text files with mathematica
> 
> Are you on a 64bit machine?
> Otherwise at 4GB you do not have enough memory even with virtual memory
> to read the file. It was created because it was written by small
> segments.
> Make sure you only try to read it small chunks at a time.
> 
> -----Original Message-----
> From: Andrey Shevchuk [mailto:AShevchuk at Maplebank.com] 
To: mathgroup at smc.vnet.net
> Sent: Saturday, April 23, 2005 7:16 AM
> Subject: [mg56439] [mg56378] Working with huge text files with mathematica
> 
> 
> Hi Everybody,
> 
> I face a problem when try to read in a huge data file with my
> Mathematica5.1
> 
> The file is approx. 4Gb large and was created by another Mathematica
> application (So, I think Mathematica should be able to handle it). 
> 
> Now, if I try to use Read on a stream from this file and it returns
> EndOfFile and nothing else!
> I checked the file with the HexEdit (an editor for huge files) and it is
> not
> corrupt and has actually the data I need.
> 
> A similar file (from the same application) but 1Gb works perfectly.
> Any ideas?
> 
> Does Mathematica have an internal limit for the file size (and if yes,
> can
> one somehow override this option) ?
> 
> I would appreciate any feedback!
> 
> 


  • Prev by Date: "large" matrices, Eigenvalues, determinants, characteristic polynomials
  • Next by Date: Re: Working with huge text files with mathematica
  • Previous by thread: Re: Working with huge text files with mathematica
  • Next by thread: Re: Working with huge text files with mathematica