Re: Memory problem when multiplying large sparse matrices...
- To: mathgroup at smc.vnet.net
- Subject: [mg70763] Re: Memory problem when multiplying large sparse matrices...
- From: David Bailey <dave at Remove_Thisdbailey.co.uk>
- Date: Thu, 26 Oct 2006 02:39:20 -0400 (EDT)
- References: <ehkbs1$g5c$1@smc.vnet.net>
renormalize at hotmail.com wrote:
> I encounter a frustrating memory problem when using Mathematica 5.2 to
> multiply one large sparse matrix by another diagonal one:
>
> In[1]: = $HistoryLength=0;
>
> In[2]: = bigmat = Import[ "C:\Documents and Settings/bigmat.mx",
> "Dump"]
>
> Out[2] = SparseArray[<48356957>, {38743, 76936}]
>
> In[3]: = diagmat = Import["C:\Documents and Settings/diagmat.mx",
> "Dump"]
>
> Out[3] = SparseArray[<38743>, {38743, 38743}]
>
> In[4]: = {ByteCount[bigmat], ByteCount[diagmat], MemoryInUse[ ],
> MaxMemoryUsed[ ]}
>
> Out[4] = {387011004, 1085152, 391366120, 391528736}
>
> In[5]: = prodmat = diagmat . bigmat
>
>>From In[5] :=
>
> No more memory available.
> Mathematica kernel has shut down.
> Try quitting other applications and then retry.
>
> The out-of-memory message appears almost immediately upon executing
> In[5]. Now, my Windows XP machine has 4 GB of RAM, and since I don't
> have the \3GB switch set, I assume that Mathematica can access only 2
> GB. But 2 GB seems like more than enough memory for this problem.
> After all, bigmat occupies 387 MB, and the resulting prodmat should
> require only another 387 MB. So what is it about performing the matrix
> dot product in In[5] that instantly balloons memory beyond the 2 GB
> limit?
>
> Puzzled,
> Ron
>
If your sparse matrix occupies 387MB in its sparse form it must have
enormous dimensions and/or not be very sparse! Remember that a sparse
array can be bigger than the equivalent 'ordinary' array if it is not in
fact very sparse. I think it is inevitable that intermediate steps in
this multiplication will require more memory - for example, at the very
least, the result will be accumulated while the original is still in
memory. Furthermore, if you are doing a calculation of this size, the
chances are that you have already used considerable memory to store
other objects.
I suggest you either represent diagmat as a vector and perform the
multiplication explicitly or get yourself a 64-bit machine with a lot of
physical memory!
David Bailey
http://www.dbaileyconsultancy.co.uk