RE: Re: how to quickly read a >10MB big file
- To: mathgroup at smc.vnet.net
- Subject: [mg72900] RE: [mg72865] Re: how to quickly read a >10MB big file
- From: "David Annetts" <davidannetts at aapt.net.au>
- Date: Wed, 24 Jan 2007 17:29:34 -0500 (EST)
- References: <200701200844.DAA10812@smc.vnet.net><eoveue$rlt$1@smc.vnet.net>
Hi, > Sorry, I forgot to tell you the details. Let me give them now. > > The format of the file is that five note lines followed by a > block of data (6 columns * 100000 lines). It looks like as below: > -------------------------------------------------------------- > --------------------------------------- > The file was generated on Jan-01-2007 > > ParameterA=0.20998977 > ParameterB=-2323.898780 > ParameterC=1223 > > the full output is: > > -7.9777019460E-03 5.8979296313E-03 -5.8992690654E-02 > -1.9555038170E-03 -0.2143438800 0.9835566699 > 9.5788225640E-02 -1.6666155312E-02 -2.3570413269E-02 > 8.4937134986E-04 -0.1289696421 0.9813171342 > 6.7266728621E-02 -2.7685295289E-02 4.8717250310E-02 > 1.5101454940E-02 -0.1758737132 0.9917945596 > ... > ... > ... > -------------------------------------------------------------- > --------------------------------------- > > My PC has a Pentium4 CPU and 512MB memory. I have used > "Import" (using type Table), "ReadList" and "FindList", but > all of them were very slow. I assume those 6 numbers are all on one line? I'd read this using one of two methods .... 1. ReadList[] as records; Drop[] the first five; pick off the numbers using ToExpression[StringSplit[#]]& /@ (remainder of data) Or 2. OpenRead[] your file; Skip[] the first 5 records; ReadList[] the remainder of the data as arrays of numbers; Close[] the file. I suspect option 2 will work better. Regards, Dave.
- References:
- how to quickly read a >10MB big file
- From: "P.Wholey" <p.wholey@gmail.com>
- how to quickly read a >10MB big file