MathGroup Archive 2010

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: convert txt to binary file

  • To: mathgroup at smc.vnet.net
  • Subject: [mg108962] Re: convert txt to binary file
  • From: Bill Rowe <readnews at sbcglobal.net>
  • Date: Thu, 8 Apr 2010 08:02:40 -0400 (EDT)

On 4/7/10 at 3:21 AM, jihane.ajaja at mail.mcgill.ca (jihane) wrote:

>Thank you for all your replies. To give more details about my file:
>it is a file with numerical data, presented in 3 columns for x y and
>z axis. (they are acceleration measurements). My computer is a 64
>bits machine with 8 GB of RAM. Why my file is that huge? well the
>measurements are done 1000 times per second. I can't ignore parts of
>the data. I need to analyze all of it. I don't think of any other
>useful detail I could provide. What I want to do with this data is
>do some basic calculations and generate a couple of plots.

A few thoughts. Your other message indicated each value in your
file is represented with ~16 characters including the character
that acts as the column separator. A bit of experimentation on
my system (also a 64 bit system) with ByteCount indicates
Mathematica stores a single real value using 16 bytes. So, the
4GB file you have will occupy a bit more than 4GB of internal
memory since I assume the data will be in an array. Mathematica
has some overhead associated with internal storage of arrays in
addition to the storage for each element of the array.

Since this is more than half the RAM you have available, it is
clear any operation that would make a copy of the array will
fail due to insufficient memory. In fact, when you consider RAM
needed by your operating system, the Mathematica program and
anything else you might have running, it is pretty easy to see
there won't be much you can do with that much data read into RAM.

=46urther, even if you could actually create a simple time plot of
all the data after it is read into RAM, the resolution of your
screen and printer is not sufficient to display that much data
at once. For example, my system has a horizontal display width
of 1280 pixels. If I were to plot a data array with say 10,000
data points, it is clear some of those points must overlap. And
note, 10,000 real triplets would need less than ~500 KB to
store. So, 10,000 data points is clearly a small subset of your data.

Your only real choices to work with as much data as you've
described, are either to intelligently downsample the data or
break the data up into smaller blocks and work with those. And
note, this really isn't a limitation Mathematica imposes. It is
an inherent limitation of the amount of RAM you have available
and real display sizes.



  • Prev by Date: Re: Inverse Laplace Transform
  • Next by Date: Re: Branching on error messages?
  • Previous by thread: Re: convert txt to binary file
  • Next by thread: Re: convert txt to binary file