Memory conserving Input of big Ascii tables ?
- To: mathgroup at smc.vnet.net
- Subject: [mg35400] Memory conserving Input of big Ascii tables ?
- From: heimann at ism.tu-berlin.de (Justus Heimann)
- Date: Thu, 11 Jul 2002 05:24:05 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
Hello,
I'm seeking for a memory (i.e., Mathematica kernel and WindowsOS
memory!) conserving way to read in big Ascii tables (in fact quadratic
matrices of up to 10.000 x 10.000 elements) of double precision real
numbers ? The processing time is of minor importance, memory saving is
the key!
What I tried with my Mathematica 4.0 Windows version is a simple
sequentially procedure like:
----------
funcReadSeqBigMatrices[filein_String]:=
Module[{streamIn, DUMMY, MATRIX = {}},
streamIn = OpenRead[filein];
Do[{
DUMMY =
ReadList[streamIn, Array[Number&, 3], Ni][[All, 3]];
AppendTo[MATRIX, DUMMY];
Clear[DUMMY]; Unprotect[Out]; Clear[Out]; Protect[Out];
}, {Nj}];
Close[streamIn];
MATRIX = Developer`ToPackedArray[MATRIX, Real];
inMATRIX := Return[MATRIX];
];
----------
in order to read in Ascii tables formated like (row index i, column
index j, value) of the form:
----------
1 1 0.529597998598605E-04
1 2 0.530631874495232E-04
1 ..... .....................
1 10000 0.516990975201977E-04
2 1 0.529597998598605E-04
2 2 0.530631874495232E-04
2 ..... .....................
2 10000 0.532446901533145E-04
..... ..... .....................
10000 1 0.532674217089594E-04
10000 2 0.517207876590374E-04
10000 ..... .....................
10000 10000 0.516990975201977E-04
----------
But, this isn't memory conserving enough in my case. Does anyone know
a nice and memory conserving way to do this job ???
Many many thanks!
Justus