MathGroup Archive 2007

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: how to quickly read a >10MB big file

  • To: mathgroup at
  • Subject: [mg72865] Re: how to quickly read a >10MB big file
  • From: "P.Wholey" <p.wholey at>
  • Date: Tue, 23 Jan 2007 04:51:14 -0500 (EST)
  • References: <><eoveue$rlt$>

Thanks to all!

Sorry, I forgot to tell you the details. Let me give them now.

The format of the file is that five note lines followed by a block of
data (6 columns * 100000 lines). It looks like as below:
The file was generated on Jan-01-2007


the full output is:

 -7.9777019460E-03  5.8979296313E-03 -5.8992690654E-02
-1.9555038170E-03 -0.2143438800      0.9835566699
  9.5788225640E-02 -1.6666155312E-02 -2.3570413269E-02
8.4937134986E-04 -0.1289696421      0.9813171342
  6.7266728621E-02 -2.7685295289E-02  4.8717250310E-02
1.5101454940E-02 -0.1758737132      0.9917945596

My PC has a Pentium4 CPU and 512MB memory. I have used "Import" (using
type Table), "ReadList" and "FindList", but all of them were very slow.

Looking forward to your help!


David Annetts wrote:
> Hi,
> > Hello, Friends:
> > It is a file which has nearly 100,000 data lines. The file size is
> > >10MB. Using mathematica 5, it takes over 30 mins to read the
> > data in; but using fortran, it takes only several secs. Does
> > anyone has a good idea to read such a file quickly. Many
> > thanks for any help in advance!
> How are you reading the file into Mathematica?  Read?  ReadList?  Import?
> The MathGroup package FastBinary?  Are you reading the file, then processing
> it, or processing it as you go?
> All these factors will have a bearing on the speed that you can read your
> data into Mathematica.  In addition, while Fortran can be quite a bit
> quicker than Mathematica, are you sure it's a fair comparison between the
> two?
> Please tell the list a bit more about how you're working.  Posting code and
> a sample of the data you're working with will probably get quicker, more
> appropriate answers than not.
> Regards,
> Dave.

  • Prev by Date: Re: pursuit curve (differential equations)
  • Next by Date: Finding bounds of the domain of a Piecewise function
  • Previous by thread: Re: how to quickly read a >10MB big file
  • Next by thread: RE: Re: how to quickly read a >10MB big file