[Date Index]
[Thread Index]
[Author Index]
Re: convert txt to binary file
*To*: mathgroup at smc.vnet.net
*Subject*: [mg108954] Re: convert txt to binary file
*From*: David Bailey <dave at removedbailey.co.uk>
*Date*: Thu, 8 Apr 2010 08:01:11 -0400 (EDT)
*References*: <hphbpe$lll$1@smc.vnet.net>
jihane wrote:
> Thank you for all your replies.
> To give more details about my file: it is a file with numerical data, presented in 3 columns for x y and z axis. (they are acceleration measurements). My computer is a 64 bits machine with 8 GB of RAM. Why my file is that huge? well the measurements are done 1000 times per second. I can't ignore parts of the data. I need to analyze all of it. I don't think of any other useful detail I could provide. What I want to do with this data is do some basic calculations and generate a couple of plots.
>
> Thank you again for all the great help!
>
OK, well if you want Mathematica to run in 64 bit mode (under Windows)
you need to load the 64-bit version of Windows on your machine. This
will then run both 64 and 32 bit applications. When you install
Mathematica it will recognise this configuration and load the 64-bit
version. Together with your 8GB or RAM, you will be able to process much
larger problems with Mathematica.
Even so, that is a lot of data, and if you plot it all at once, you will
have an absurd number of plot points. I can imagine that perhaps you
want to plot it in sections, or maybe smooth it and plot the result. In
either case, you can read it in sections using:
stream=OpenRead[file];
followed by:
ReadList[stream,Real,nn]
where nn is the number of lines you want to read in one block. I reckon
you have approx 250 million numbers in that file, and at 8 bytes each,
that will be 2GB of storage to hold the result, so it is not impossible
that you could store that many numbers, but your memory might run short
while reading the data in.
Be aware also, that lists of numbers can be stored in two ways in
Mathematica - packed and unpacked. They look the same, but packed arrays
are much more efficient, and occupy less storage. I tried a little
experiment, and ReadList applied to your data seems to return an
unpacked array, so you may want to read it in with:
Developer`ToPackedArray[ReadList[stream,Real,nn]]
Unless you intend to read the data in over and over again, there isn't
going to be too much point in converting the file to binary format.
My advice would be to start with a small portion of the file - either a
small time range, or every N'th sample, as appropriate, and develop your
code using that small file - bearing in mind that you will need to scale
the result up later.
I suggest you describe what you want to do with that data - would it be
easy to handle it in blocks - because that would be the obvious approach.
David Bailey
http://www.dbaileyconsultancy.co.uk
Prev by Date:
**Re: Branching on error messages?**
Next by Date:
**Re: convert txt to binary file**
Previous by thread:
**Re: convert txt to binary file**
Next by thread:
**Re: convert txt to binary file**
| |