MathGroup Archive 2006

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Memory problem when multiplying large sparse matrices...

  • To: mathgroup at
  • Subject: [mg70763] Re: Memory problem when multiplying large sparse matrices...
  • From: David Bailey <dave at>
  • Date: Thu, 26 Oct 2006 02:39:20 -0400 (EDT)
  • References: <ehkbs1$g5c$>

renormalize at wrote:
> I encounter a frustrating memory problem when using Mathematica 5.2 to
> multiply one large sparse matrix by another diagonal one:
> In[1]: = $HistoryLength=0;
> In[2]: = bigmat = Import[ "C:\Documents and Settings/",
> "Dump"]
> Out[2] = SparseArray[<48356957>, {38743, 76936}]
> In[3]: = diagmat = Import["C:\Documents and Settings/",
> "Dump"]
> Out[3] = SparseArray[<38743>, {38743, 38743}]
> In[4]: = {ByteCount[bigmat], ByteCount[diagmat], MemoryInUse[ ],
> MaxMemoryUsed[ ]}
> Out[4] = {387011004, 1085152, 391366120, 391528736}
> In[5]: = prodmat = diagmat . bigmat
>>From In[5] :=
> No more memory available.
> Mathematica kernel has shut down.
> Try quitting other applications and then retry.
> The out-of-memory message appears almost immediately upon executing
> In[5].  Now, my Windows XP machine has 4 GB of RAM, and since I don't
> have the \3GB switch set, I assume that Mathematica can access only 2
> GB.  But 2 GB seems like more than enough memory for this problem.
> After all, bigmat occupies 387 MB, and the resulting prodmat should
> require only another 387 MB.  So what is it about performing the matrix
> dot product in In[5] that instantly balloons memory beyond the 2 GB
> limit?
> Puzzled,
> Ron
If your sparse matrix occupies 387MB in its sparse form it must have 
enormous dimensions and/or not be very sparse! Remember that a sparse 
array can be bigger than the equivalent 'ordinary' array if it is not in 
fact very sparse. I think it is inevitable that intermediate steps in 
this multiplication will require more memory - for example, at the very 
least, the result will be accumulated while the original is still in 
memory. Furthermore, if you are doing a calculation of this size, the 
chances are that you have already used considerable memory to store 
other objects.

I suggest you either represent diagmat as a vector and perform the 
multiplication explicitly or get yourself a 64-bit machine with a lot of 
physical memory!

David Bailey

  • Prev by Date: MathModelica System Designer Professional now available
  • Next by Date: Re: "Declaring" a vector for NDSolve
  • Previous by thread: Memory problem when multiplying large sparse matrices...
  • Next by thread: Evaluate, /., {{...}}, etc.