Services & Resources / Wolfram Forums
-----
 /
MathGroup Archive
2002
*January
*February
*March
*April
*May
*June
*July
*August
*September
*October
*November
*December
*Archive Index
*Ask about this page
*Print this page
*Give us feedback
*Sign up for the Wolfram Insider

MathGroup Archive 2002

[Date Index] [Thread Index] [Author Index]

Search the Archive

Memory conserving Input of big Ascii tables ?

  • To: mathgroup at smc.vnet.net
  • Subject: [mg35400] Memory conserving Input of big Ascii tables ?
  • From: heimann at ism.tu-berlin.de (Justus Heimann)
  • Date: Thu, 11 Jul 2002 05:24:05 -0400 (EDT)
  • Sender: owner-wri-mathgroup at wolfram.com

Hello,

I'm seeking for a memory (i.e., Mathematica kernel and WindowsOS
memory!) conserving way to read in big Ascii tables (in fact quadratic
matrices of up to 10.000 x 10.000 elements) of double precision real
numbers ? The processing time is of minor importance, memory saving is
the key!

What I tried with my Mathematica 4.0 Windows version is a simple
sequentially procedure like:

----------
funcReadSeqBigMatrices[filein_String]:=
   Module[{streamIn, DUMMY, MATRIX = {}},
      
         streamIn = OpenRead[filein];

         Do[{
             DUMMY = 
             ReadList[streamIn, Array[Number&, 3], Ni][[All, 3]];
             AppendTo[MATRIX, DUMMY];
             Clear[DUMMY]; Unprotect[Out]; Clear[Out]; Protect[Out];
             
         }, {Nj}];

         Close[streamIn];
         MATRIX = Developer`ToPackedArray[MATRIX, Real];
         inMATRIX := Return[MATRIX];
   ];
----------

in order to read in Ascii tables formated like (row index i, column
index j, value) of the form:

----------
    1       1   0.529597998598605E-04
    1       2   0.530631874495232E-04
    1   .....   .....................
    1   10000   0.516990975201977E-04
    2       1   0.529597998598605E-04
    2       2   0.530631874495232E-04
    2   .....   .....................
    2   10000   0.532446901533145E-04
.....   .....   .....................
10000       1   0.532674217089594E-04
10000       2   0.517207876590374E-04
10000   .....   .....................
10000   10000   0.516990975201977E-04
----------

But, this isn't memory conserving enough in my case. Does anyone know
a nice and memory conserving way to do this job ???

Many many thanks!
Justus


  • Prev by Date: = versus := and trying to speed up calculations
  • Next by Date: Integration yields different results... Why?
  • Previous by thread: Re: = versus := and trying to speed up calculations
  • Next by thread: Re: Memory conserving Input of big Ascii tables ?