Re: convert txt to binary file
- To: mathgroup at smc.vnet.net
- Subject: [mg108954] Re: convert txt to binary file
- From: David Bailey <dave at removedbailey.co.uk>
- Date: Thu, 8 Apr 2010 08:01:11 -0400 (EDT)
- References: <hphbpe$lll$1@smc.vnet.net>
jihane wrote: > Thank you for all your replies. > To give more details about my file: it is a file with numerical data, presented in 3 columns for x y and z axis. (they are acceleration measurements). My computer is a 64 bits machine with 8 GB of RAM. Why my file is that huge? well the measurements are done 1000 times per second. I can't ignore parts of the data. I need to analyze all of it. I don't think of any other useful detail I could provide. What I want to do with this data is do some basic calculations and generate a couple of plots. > > Thank you again for all the great help! > OK, well if you want Mathematica to run in 64 bit mode (under Windows) you need to load the 64-bit version of Windows on your machine. This will then run both 64 and 32 bit applications. When you install Mathematica it will recognise this configuration and load the 64-bit version. Together with your 8GB or RAM, you will be able to process much larger problems with Mathematica. Even so, that is a lot of data, and if you plot it all at once, you will have an absurd number of plot points. I can imagine that perhaps you want to plot it in sections, or maybe smooth it and plot the result. In either case, you can read it in sections using: stream=OpenRead[file]; followed by: ReadList[stream,Real,nn] where nn is the number of lines you want to read in one block. I reckon you have approx 250 million numbers in that file, and at 8 bytes each, that will be 2GB of storage to hold the result, so it is not impossible that you could store that many numbers, but your memory might run short while reading the data in. Be aware also, that lists of numbers can be stored in two ways in Mathematica - packed and unpacked. They look the same, but packed arrays are much more efficient, and occupy less storage. I tried a little experiment, and ReadList applied to your data seems to return an unpacked array, so you may want to read it in with: Developer`ToPackedArray[ReadList[stream,Real,nn]] Unless you intend to read the data in over and over again, there isn't going to be too much point in converting the file to binary format. My advice would be to start with a small portion of the file - either a small time range, or every N'th sample, as appropriate, and develop your code using that small file - bearing in mind that you will need to scale the result up later. I suggest you describe what you want to do with that data - would it be easy to handle it in blocks - because that would be the obvious approach. David Bailey http://www.dbaileyconsultancy.co.uk