[Date Index] [Thread Index] [Author Index]

Re: Working with huge text files with mathematica

• To: mathgroup at smc.vnet.net
• Subject: [mg56463] Re: [mg56378] Working with huge text files with mathematica
• From: yehuda ben-shimol <bsyehuda at gmail.com>
• Date: Tue, 26 Apr 2005 01:33:25 -0400 (EDT)
• References: <200504230516.BAA03095@smc.vnet.net>
• Reply-to: yehuda ben-shimol <bsyehuda at gmail.com>
• Sender: owner-wri-mathgroup at wolfram.com

I wonder under which operating system you are working. 
From the limitations you mentioned I assume that it is MS Windows
which has a 2GB limitation (currently), and I suspect that writing the
data into the file was done gradually and not in once. I'm I'm right
in my speculations then working with unix/linux can solve it. I was
working (in the past) with a Linux system using 990 MB out of 1GB RAM
and 3.9GB out of 4.0 GB of swap file and the system was stable for
weeks with no porblems.
yehuda

On 4/23/05, Andrey Shevchuk <AShevchuk at maplebank.com> wrote:
> 
> Hi Everybody,
> 
> I face a problem when try to read in a huge data file with my Mathematica5.1
> 
> The file is approx. 4Gb large and was created by another Mathematica
> application (So, I think Mathematica should be able to handle it).
> 
> Now, if I try to use Read on a stream from this file and it returns
> EndOfFile and nothing else!
> I checked the file with the HexEdit (an editor for huge files) and it is not
> corrupt and has actually the data I need.
> 
> A similar file (from the same application) but 1Gb works perfectly.
> Any ideas?
> 
> Does Mathematica have an internal limit for the file size (and if yes, can
> one somehow override this option) ?
> 
> I would appreciate any feedback!
> 
>

• References:
  • Working with huge text files with mathematica
    • From: Andrey Shevchuk <AShevchuk@Maplebank.com>