Re: IO-Performance of Mathematica 4.1

*To*: mathgroup at smc.vnet.net*Subject*: [mg44787] Re: IO-Performance of Mathematica 4.1*From*: "Christos Argyropoulos M.D." <argchris at otenet.gr>*Date*: Thu, 27 Nov 2003 11:38:12 -0500 (EST)*References*: <bphtib$1lu$1@smc.vnet.net>*Sender*: owner-wri-mathgroup at wolfram.com

Hello, Ran into the same problem with really big datasets (Iam using Mathematica to analyze measurements from "DNA-chips", which tend to be huge files). One can speed mathematica's I/O by Importing numerical data as "Table" and setting ConversiongOptions->{DataBlockErrorChecking" ->False} Christos Argyropoulos MD Patras Greece ----- Original Message ----- From: "Marko Kastens" <Kastens at Hamburg.baw.de> To: mathgroup at smc.vnet.net Subject: [mg44787] IO-Performance of Mathematica 4.1 > Hi! > > I'am searching for a good (fast) way to import big datasets in Mathematica 4.1. > > What I've done so far: > - The read-in from the _orignal_ (ASCII)file needs 97 sec. (a 5xxxx x 2 > matrix) > - Exporting my matrix using Export[] with "List" or "Table" needs > (nearly) infinity of time :-( > - Exporting the matrix with the HDF (Hierarchy Data Format): 7 seconds > :-) Great, but... > - Importing the matrix with HDF: fast but wrong: the HDF seems to > supports only Real64. My first column is a 10-digit-Integer representing > the MM-time-format. By converting this to yyExx the last digits get lost > and my time is doing funny jumps... > > - using the Experimental-Functions like this: > > << Experimental` > t0 = AbsoluteTime[]; > strm = OpenWrite["tmp0.bin", DOSTextFormat -> False]; > BinaryExport[strm, Length[data], "Integer32"] > For[i = 1, i < Length[data] + 1, > { > BinaryExport[strm, data[[i]], {"Integer64", "Real64", "Integer64"}] > }; i++] > Close[strm]; > Print[AbsoluteTime[] - t0]; > > it takes 415 seconds to export and 247 seconds to import. Even worser > than importing the original. > > Only the good old ASCII-way is better: > > t0 = AbsoluteTime[]; > strm = OpenWrite["tmp1.dat"]; > Write[strm, Length[data]] > For[i = 1, i < Length[data] + 1, > { > Write[strm, data[[i, 1]]], > Write[strm, data[[i, 2]]], > Write[strm, data[[i, 3]]] > }; i++] > Close[strm]; > Print[AbsoluteTime[] - t0]; > > It takes 32 seconds to export and 16 seconds to import. > > > Hmm, normaly binary-import/export is faster than ascii-import/export. > Does anybody know a proper and fast method? Thanks a lot. > > marko >