MathGroup Archive 2009

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: accessing data in a large file

  • To: mathgroup at smc.vnet.net
  • Subject: [mg95842] Re: accessing data in a large file
  • From: Bill Rowe <readnews at sbcglobal.net>
  • Date: Wed, 28 Jan 2009 06:33:33 -0500 (EST)

On 1/27/09 at 6:58 AM, wb at wavebounce.com wrote:

>I'm trying to get data from a 450MB file. I can get the first 300MB
>(or 150 000 000 Integer16) with the code below but any more than
>that causes Mathematica 7 to shut down with out of memory error
>(WinXP, SP3, 2GB RAM).

>n = 150000000; inFile = OpenRead["DR.wav", BinaryFormat -> True]; y
>= BinaryReadList[inFile, "Integer16", n]; Close[inFile];

>My reading of the documentation indicates that the only option I
>have with BinaryReadList is to read from the start of the file to
>depth n (negative values of n it won't accept). If I could start
>reading from some point into the file, I could split the file up
>into pieces by exporting segments to other files.

You can easily read portions of a binary file by opening it
before using BinaryReadList.

That is doing

str=Open[filename,BinaryFormat->True];
data=BinaryReadList[str,"Integer16",200];

will read the first 400 bytes as 16 bit integers and leave the
file open set to read the next chunk of data. You can determine
where you are in the file using StreamPosition. And you can move
to an arbitrary position in the file using SetStreamPosition.
All you need to do when you finish is close the file using Close.



  • Prev by Date: Re: Permutations...
  • Next by Date: Re: Has anyone tried to use MathLink to receive real-time market data
  • Previous by thread: Re: accessing data in a large file
  • Next by thread: Re: accessing data in a large file