Re: Memory problem when multiplying large sparse matrices...

*To*: mathgroup at smc.vnet.net*Subject*: [mg70763] Re: Memory problem when multiplying large sparse matrices...*From*: David Bailey <dave at Remove_Thisdbailey.co.uk>*Date*: Thu, 26 Oct 2006 02:39:20 -0400 (EDT)*References*: <ehkbs1$g5c$1@smc.vnet.net>

renormalize at hotmail.com wrote: > I encounter a frustrating memory problem when using Mathematica 5.2 to > multiply one large sparse matrix by another diagonal one: > > In[1]: = $HistoryLength=0; > > In[2]: = bigmat = Import[ "C:\Documents and Settings/bigmat.mx", > "Dump"] > > Out[2] = SparseArray[<48356957>, {38743, 76936}] > > In[3]: = diagmat = Import["C:\Documents and Settings/diagmat.mx", > "Dump"] > > Out[3] = SparseArray[<38743>, {38743, 38743}] > > In[4]: = {ByteCount[bigmat], ByteCount[diagmat], MemoryInUse[ ], > MaxMemoryUsed[ ]} > > Out[4] = {387011004, 1085152, 391366120, 391528736} > > In[5]: = prodmat = diagmat . bigmat > >>From In[5] := > > No more memory available. > Mathematica kernel has shut down. > Try quitting other applications and then retry. > > The out-of-memory message appears almost immediately upon executing > In[5]. Now, my Windows XP machine has 4 GB of RAM, and since I don't > have the \3GB switch set, I assume that Mathematica can access only 2 > GB. But 2 GB seems like more than enough memory for this problem. > After all, bigmat occupies 387 MB, and the resulting prodmat should > require only another 387 MB. So what is it about performing the matrix > dot product in In[5] that instantly balloons memory beyond the 2 GB > limit? > > Puzzled, > Ron > If your sparse matrix occupies 387MB in its sparse form it must have enormous dimensions and/or not be very sparse! Remember that a sparse array can be bigger than the equivalent 'ordinary' array if it is not in fact very sparse. I think it is inevitable that intermediate steps in this multiplication will require more memory - for example, at the very least, the result will be accumulated while the original is still in memory. Furthermore, if you are doing a calculation of this size, the chances are that you have already used considerable memory to store other objects. I suggest you either represent diagmat as a vector and perform the multiplication explicitly or get yourself a 64-bit machine with a lot of physical memory! David Bailey http://www.dbaileyconsultancy.co.uk