Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

RE: Re: how to quickly read a >10MB big file

• To: mathgroup at smc.vnet.net
• Subject: [mg72918] RE: [mg72865] Re: how to quickly read a >10MB big file
• From: "David Annetts" <davidannetts at aapt.net.au>
• Date: Thu, 25 Jan 2007 06:47:25 -0500 (EST)
• References: <200701200844.DAA10812@smc.vnet.net><eoveue$rlt$1@smc.vnet.net>

Hi All,

It was late here last night when I replied.  Apologies for the lack of
examples in my post.

> Thanks to all!
> Sorry, I forgot to tell you the details. Let me give them now.
> 
> The format of the file is that five note lines followed by a 
> block of data (6 columns * 100000 lines). It looks like as below:
> --------------------------------------------------------------
> ---------------------------------------
> The file was generated on Jan-01-2007
> 
> ParameterA=0.20998977
> ParameterB=-2323.898780
> ParameterC=1223
> 
> the full output is:
> 
>  -7.9777019460E-03  5.8979296313E-03 -5.8992690654E-02
> -1.9555038170E-03 -0.2143438800      0.9835566699
>   9.5788225640E-02 -1.6666155312E-02 -2.3570413269E-02
> 8.4937134986E-04 -0.1289696421      0.9813171342
>   6.7266728621E-02 -2.7685295289E-02  4.8717250310E-02
> 1.5101454940E-02 -0.1758737132      0.9917945596
> ...
> ...
> ...
> --------------------------------------------------------------
> ---------------------------------------
> 
> My PC has a Pentium4 CPU and 512MB memory. I have used 
> "Import" (using type Table), "ReadList" and "FindList", but 
> all of them were very slow.
> 
> Looking forward to your help!

On my system (P4 with 1 GB RAM running @ 3.2 GHz with speed issues fixed) I
get 

In[34]:=
Timing[r01 = Import["Wholey.txt", "Table"];]
Out[34]=
{7.282 Second,Null}

And 

In[35]:=
Timing[
  inp = OpenRead["Wholey.txt"];
  Skip[inp, Record, 5];
  r02 = ReadList[inp, Table[Number, {6}]];
  Close[inp];
  ]

Out[35]=
{6.796 Second,Null}

With 

In[29]:=
SameQ[Drop[r01, {1, 5}], r02]

Out[29]=
True

For an example based on your specs above

In[31]:=
SetDirectory["d:/Tmpfiles"]
FileNames["*.txt"]
FileInformation[First@%]

Out[31]=
d:\Tmpfiles

Out[32]=
{Wholey.txt}

Out[33]=
{File\[Rule]d:\Tmpfiles\Wholey.txt,FileType\[Rule]File,Date\[Rule]3378660417
,\
ByteCount\[Rule]11616729}

Reading >10 MB files in under 10 s seems reasonable to me, although I can
read the file in <1 s using F90.  What else is running on your system? 

Regards,

Dave. 

• References:
  • how to quickly read a >10MB big file
    • From: "P.Wholey" <p.wholey@gmail.com>