[Date Index] [Thread Index] [Author Index]

RE: Re: how to quickly read a >10MB big file

• To: mathgroup at smc.vnet.net
• Subject: [mg72900] RE: [mg72865] Re: how to quickly read a >10MB big file
• From: "David Annetts" <davidannetts at aapt.net.au>
• Date: Wed, 24 Jan 2007 17:29:34 -0500 (EST)
• References: <200701200844.DAA10812@smc.vnet.net><eoveue$rlt$1@smc.vnet.net>

Hi,
> Sorry, I forgot to tell you the details. Let me give them now.
> 
> The format of the file is that five note lines followed by a 
> block of data (6 columns * 100000 lines). It looks like as below:
> --------------------------------------------------------------
> ---------------------------------------
> The file was generated on Jan-01-2007
> 
> ParameterA=0.20998977
> ParameterB=-2323.898780
> ParameterC=1223
> 
> the full output is:
> 
>  -7.9777019460E-03  5.8979296313E-03 -5.8992690654E-02
> -1.9555038170E-03 -0.2143438800      0.9835566699
>   9.5788225640E-02 -1.6666155312E-02 -2.3570413269E-02
> 8.4937134986E-04 -0.1289696421      0.9813171342
>   6.7266728621E-02 -2.7685295289E-02  4.8717250310E-02
> 1.5101454940E-02 -0.1758737132      0.9917945596
> ...
> ...
> ...
> --------------------------------------------------------------
> ---------------------------------------
> 
> My PC has a Pentium4 CPU and 512MB memory. I have used 
> "Import" (using type Table), "ReadList" and "FindList", but 
> all of them were very slow.

I assume those 6 numbers are all on one line?

I'd read this using one of two methods ....

1. ReadList[] as records; Drop[] the first five; pick off the numbers using
ToExpression[StringSplit[#]]& /@ (remainder of data)

Or 

2. OpenRead[] your file; Skip[] the first 5 records; ReadList[] the
remainder of the data as arrays of numbers; Close[] the file.

I suspect option 2 will work better.

Regards,

Dave.

• References:
  • how to quickly read a >10MB big file
    • From: "P.Wholey" <p.wholey@gmail.com>