Re: Efficient way of reading matrices
- To: mathgroup at smc.vnet.net
- Subject: [mg85329] Re: Efficient way of reading matrices
- From: m.r at inbox.ru
- Date: Wed, 6 Feb 2008 05:40:12 -0500 (EST)
- References: <fo3gad$b3p$1@smc.vnet.net> <fo9h1r$t26$1@smc.vnet.net>
On Feb 5, 5:24=A0am, "Szabolcs Horv=E1t" <szhor... at gmail.com> wrote:
> In Feb 5, 2008 7:22 AM, =A0<m... at inbox.ru> wrote:
>
>
>
> > Hi Szabolcs,
>
> > maybe you can read and pack your matrix row by row:
>
> > WriteString["/zzz", Sequence @@ Riffle[#, " "], "\n"]& /@
> > =A0 RandomInteger[{-1000, 1000}, {200, 250000}]
> > Close["/zzz"]
> > =A0(*a test file*)
>
> > (m = Developer`ToPackedArray@ Read["/zzz",
> > =A0 Array[Number&, 250000]]& /@ Range[200];) // Timing
> > Close["/zzz"]
>
> Thanks you Maxim Rytin,
>
> Good and useful suggestion, as always. :-) =A0I have already written a
> MathLink matrix reader program to do the work, but your program
> (though slower) uses even less memory as it never needs to store two
> copies of the complete matrix in memory.
>
> Szabolcs Horv=E1t
The speed can be improved some because a lot of time is spent
evaluating Number&[arrayindex]:
In[1]:= (m = Developer`ToPackedArray@ Read["/zzz",
Array[Number&, 250000]]& /@ Range[200];) // Timing
Out[1]= {50.204, Null}
In[1]:= Module[{f = ConstantArray[Number, 250000]},
m = Developer`ToPackedArray@ Read["/zzz",
f]& /@ Range[200]]; // Timing
Out[1]= {22.625, Null}
Maxim Rytin
m.r at inbox.ru