Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Re: accessing data in a large file

• To: mathgroup at smc.vnet.net
• Subject: [mg95803] Re: [mg95784] accessing data in a large file
• From: "Tugrul Temel" <temelt at xs4all.nl>
• Date: Wed, 28 Jan 2009 06:26:13 -0500 (EST)
• References: <200901271158.GAA23236@smc.vnet.net>

You can use Save command (see Math Documents library) to save the
intermediate data as separate files. Basically, you divide the original data
set into different segments, and each segment is saved with a different name
such as one, two, three..so on. Once you retrieve the entire dataset, then
you can combine them as one file.

Regards,
Temel


-----Original Message-----
From: wb at wavebounce.com [mailto:wb at wavebounce.com] 
Sent: Tuesday, January 27, 2009 12:59 PM
To: mathgroup at smc.vnet.net
Subject: [mg95803] [mg95784] accessing data in a large file

I'm trying to get data from a 450MB file. I can get the first 300MB (or 150
000 000 Integer16) with the code below but any more than that causes
Mathematica 7 to shut down with out of memory error (WinXP, SP3, 2GB RAM).

n = 150000000; 
inFile = OpenRead["DR.wav", BinaryFormat -> True];
y = BinaryReadList[inFile, "Integer16", n];
Close[inFile];

My reading of the documentation indicates that the only option I have with
BinaryReadList is to read from the start of the file to depth n (negative
values of n it won't accept). If I could start reading from some point into
the file, I could split the file up into pieces by exporting segments to
other files. Also, I've found nothing in the documentation that tells me how
Mathematica 7 might use the HD to store some of this data as I read it in.
Can anyone suggest a method (however slow) that would allow me access to all
the data in this file?  Thanks.

• References:
  • accessing data in a large file
    • From: "wb@wavebounce.com" <wb@wavebounce.com>