[Date Index] [Thread Index] [Author Index]

Re: how to quickly read a >10MB big file

• To: mathgroup at smc.vnet.net
• Subject: [mg72865] Re: how to quickly read a >10MB big file
• From: "P.Wholey" <p.wholey at gmail.com>
• Date: Tue, 23 Jan 2007 04:51:14 -0500 (EST)
• References: <200701200844.DAA10812@smc.vnet.net><eoveue$rlt$1@smc.vnet.net>

Thanks to all!

Sorry, I forgot to tell you the details. Let me give them now.

The format of the file is that five note lines followed by a block of
data (6 columns * 100000 lines). It looks like as below:
-----------------------------------------------------------------------------------------------------
The file was generated on Jan-01-2007

ParameterA=0.20998977
ParameterB=-2323.898780
ParameterC=1223

the full output is:

 -7.9777019460E-03  5.8979296313E-03 -5.8992690654E-02
-1.9555038170E-03 -0.2143438800      0.9835566699
  9.5788225640E-02 -1.6666155312E-02 -2.3570413269E-02
8.4937134986E-04 -0.1289696421      0.9813171342
  6.7266728621E-02 -2.7685295289E-02  4.8717250310E-02
1.5101454940E-02 -0.1758737132      0.9917945596
...
...
...
-----------------------------------------------------------------------------------------------------

My PC has a Pentium4 CPU and 512MB memory. I have used "Import" (using
type Table), "ReadList" and "FindList", but all of them were very slow.

Looking forward to your help!

PW


David Annetts wrote:
> Hi,
>
> > Hello, Friends:
> > It is a file which has nearly 100,000 data lines. The file size is
> > >10MB. Using mathematica 5, it takes over 30 mins to read the
> > data in; but using fortran, it takes only several secs. Does
> > anyone has a good idea to read such a file quickly. Many
> > thanks for any help in advance!
>
> How are you reading the file into Mathematica?  Read?  ReadList?  Import?
> The MathGroup package FastBinary?  Are you reading the file, then processing
> it, or processing it as you go?
>
> All these factors will have a bearing on the speed that you can read your
> data into Mathematica.  In addition, while Fortran can be quite a bit
> quicker than Mathematica, are you sure it's a fair comparison between the
> two?
>
> Please tell the list a bit more about how you're working.  Posting code and
> a sample of the data you're working with will probably get quicker, more
> appropriate answers than not.
> 
> Regards,
> 
> Dave.

• References:
  • how to quickly read a >10MB big file
    • From: "P.Wholey" <p.wholey@gmail.com>