Services & Resources / Wolfram Forums
-----
 /
MathGroup Archive
2003
*January
*February
*March
*April
*May
*June
*July
*August
*September
*October
*November
*December
*Archive Index
*Ask about this page
*Print this page
*Give us feedback
*Sign up for the Wolfram Insider

MathGroup Archive 2003

[Date Index] [Thread Index] [Author Index]

Search the Archive

IO-Performance of Mathematica 4.1

  • To: mathgroup at smc.vnet.net
  • Subject: [mg44659] IO-Performance of Mathematica 4.1
  • From: Marko Kastens <Kastens at Hamburg.baw.de>
  • Date: Thu, 20 Nov 2003 03:16:25 -0500 (EST)
  • Sender: owner-wri-mathgroup at wolfram.com

Hi!

I'am searching for a good (fast) way to import big datasets in Mathematica 4.1.

What I've done so far:
- The read-in from the _orignal_ (ASCII)file needs 97 sec. (a 5xxxx x 2 
matrix)
- Exporting my matrix using Export[] with "List" or "Table" needs 
(nearly) infinity of time :-(
- Exporting the matrix with the HDF (Hierarchy Data Format): 7 seconds 
:-) Great, but...
- Importing the matrix with HDF: fast but wrong: the HDF seems to 
supports only Real64. My first column is a 10-digit-Integer representing 
the MM-time-format. By converting this to yyExx the last digits get lost 
and my time is doing funny jumps...

- using the Experimental-Functions like this:

<< Experimental`
t0 = AbsoluteTime[];
strm = OpenWrite["tmp0.bin", DOSTextFormat -> False];
BinaryExport[strm, Length[data], "Integer32"]
For[i = 1, i < Length[data] + 1,
   {
      BinaryExport[strm, data[[i]], {"Integer64", "Real64", "Integer64"}]
   }; i++]
Close[strm];
Print[AbsoluteTime[] - t0];

it takes 415 seconds to export and 247 seconds to import. Even worser 
than importing the original.

Only the good old ASCII-way is better:

t0 = AbsoluteTime[];
strm = OpenWrite["tmp1.dat"];
Write[strm, Length[data]]
For[i = 1, i < Length[data] + 1,
   {
     Write[strm, data[[i, 1]]],
     Write[strm, data[[i, 2]]],
     Write[strm, data[[i, 3]]]
     }; i++]
Close[strm];
Print[AbsoluteTime[] - t0];

It takes 32 seconds to export and 16 seconds to import.


Hmm, normaly binary-import/export is faster than ascii-import/export. 
Does anybody know a proper and fast method? Thanks a lot.

marko


  • Prev by Date: Re: Specify Step Size in NDSolve
  • Next by Date: Re: Improper integral
  • Previous by thread: Re: interpolation and plotting 3d data
  • Next by thread: Re: IO-Performance of Mathematica 4.1