[Date Index] [Thread Index] [Author Index]

Re: convert txt to binary file

• To: mathgroup at smc.vnet.net
• Subject: [mg108954] Re: convert txt to binary file
• From: David Bailey <dave at removedbailey.co.uk>
• Date: Thu, 8 Apr 2010 08:01:11 -0400 (EDT)
• References: <hphbpe$lll$1@smc.vnet.net>

jihane wrote:
> Thank you for all your replies.
> To give more details about my file: it is a file with numerical data, presented in 3 columns for x y and z axis. (they are acceleration measurements). My computer is a 64 bits machine with 8 GB of RAM. Why my file is that huge? well the measurements are done 1000 times per second. I can't ignore parts of the data. I need to analyze all of it. I don't think of any other useful detail I could provide. What I want to do with this data is do some basic calculations and generate a couple of plots. 
> 
> Thank you again for all the great help!
> 

OK, well if you want Mathematica to run in 64 bit mode (under Windows) 
you need to load the 64-bit version of Windows on your machine. This 
will then run both 64 and 32 bit applications. When you install 
Mathematica it will recognise this configuration and load the 64-bit 
version. Together with your 8GB or RAM, you will be able to process much 
larger problems with Mathematica.

Even so, that is a lot of data, and if you plot it all at once, you will 
have an absurd number of plot points. I can imagine that perhaps you 
want to plot it in sections, or maybe smooth it and plot the result. In 
either case, you can read it in sections using:

stream=OpenRead[file];

followed by:

ReadList[stream,Real,nn]

where nn is the number of lines you want to read in one block. I reckon 
you have approx 250 million numbers in that file, and at 8 bytes each, 
that will be 2GB of storage to hold the result, so it is not impossible 
that you could store that many numbers, but your memory might run short 
while reading the data in.

Be aware also, that lists of numbers can be stored in two ways in 
Mathematica - packed and unpacked. They look the same, but packed arrays 
are much more efficient, and occupy less storage. I tried a little 
experiment, and ReadList applied to your data seems to return an 
unpacked array, so you may want to read it in with:

Developer`ToPackedArray[ReadList[stream,Real,nn]]

Unless you intend to read the data in over and over again, there isn't 
going to be too much point in converting the file to binary format.

My advice would be to start with a small portion of the file - either a 
small time range, or every N'th sample, as appropriate, and develop your 
code using that small file - bearing in mind that you will need to scale 
the result up later.

I suggest you describe what you want to do with that data - would it be 
easy to handle it in blocks - because that would be the obvious approach.

David Bailey
http://www.dbaileyconsultancy.co.uk