RE: IO-Performance of Mathematica 4.1

*To*: mathgroup at smc.vnet.net*Subject*: [mg44743] RE: [mg44684] IO-Performance of Mathematica 4.1*From*: "Benedetto Bongiorno" <BBongiorno at attglobal.net>*Date*: Mon, 24 Nov 2003 00:05:29 -0500 (EST)*Reply-to*: <BBongiorno at attglobal.net>*Sender*: owner-wri-mathgroup at wolfram.com

I too have speed problems with importing large data sets. After many years of experimenting with various methods, I have concluded: 1 - Import & Export commands work best on *.csv 2 dimensional matrix files. 2 - Mathematica is not a speed demon on I/O functions. -----Original Message----- From: Marko Kastens [mailto:Kastens at Hamburg.baw.de] To: mathgroup at smc.vnet.net Subject: [mg44743] [mg44684] IO-Performance of Mathematica 4.1 Hi! I'am searching for a good (fast) way to import big datasets in Mathematica 4.1. What I've done so far: - The read-in from the _orignal_ (ASCII)file needs 97 sec. (a 5xxxx x 2 matrix) - Exporting my matrix using Export[] with "List" or "Table" needs (nearly) infinity of time :-( - Exporting the matrix with the HDF (Hierarchy Data Format): 7 seconds :-) Great, but... - Importing the matrix with HDF: fast but wrong: the HDF seems to supports only Real64. My first column is a 10-digit-Integer representing the MM-time-format. By converting this to yyExx the last digits get lost and my time is doing funny jumps... - using the Experimental-Functions like this: << Experimental` t0 = AbsoluteTime[]; strm = OpenWrite["tmp0.bin", DOSTextFormat -> False]; BinaryExport[strm, Length[data], "Integer32"] For[i = 1, i < Length[data] + 1, { BinaryExport[strm, data[[i]], {"Integer64", "Real64", "Integer64"}] }; i++] Close[strm]; Print[AbsoluteTime[] - t0]; it takes 415 seconds to export and 247 seconds to import. Even worser than importing the original. Only the good old ASCII-way is better: t0 = AbsoluteTime[]; strm = OpenWrite["tmp1.dat"]; Write[strm, Length[data]] For[i = 1, i < Length[data] + 1, { Write[strm, data[[i, 1]]], Write[strm, data[[i, 2]]], Write[strm, data[[i, 3]]] }; i++] Close[strm]; Print[AbsoluteTime[] - t0]; It takes 32 seconds to export and 16 seconds to import. Hmm, normaly binary-import/export is faster than ascii-import/export. Does anybody know a proper and fast method? Thanks a lot. marko