Mathematica 9 is now available
Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2008

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: SparseArray memory usage

  • To: mathgroup at smc.vnet.net
  • Subject: [mg87869] Re: [mg87839] SparseArray memory usage
  • From: Carl Woll <carlw at wolfram.com>
  • Date: Sat, 19 Apr 2008 03:32:51 -0400 (EDT)
  • References: <200804180641.CAA12681@smc.vnet.net>

wpb wrote:

>Hi,
>
>An obvious advantage of SparseArray is that less memory is needed for sparse arrays. Unfortunately, this doesn't always seem to be the case, as I outline below sometimes almost as much memory is used as for an ordinary array.
>
>Can someone explain this, and hopefully tell me how to get around it?
>  
>
It's because of the way SparseArrays are internally structured. It 
stores a cumulative count of the number of nonzero elements in each row. 
For a matrix with 10^8 rows, this information takes:

In[1]:= ByteCount[ConstantArray[1, 10^8]]

Out[1]= 400000076

This is the memory usage you are seeing.

Obviously, if you can structure your problem so that it has fewer rows, 
your memory usage will go down.

Carl Woll
Wolfram Research

>First, a situation which is fine:
>Starting with a clean kernel, I get the following memory usage for a 2 x 100million sparse array:
>
>In[1]:= MaxMemoryUsed[]
>MemoryInUse[]
>a=SparseArray[{{1,1}->1},{2,100000000}];
>MaxMemoryUsed[]
>MemoryInUse[]
>
>Out[1]= 5826800
>Out[2]= 6652184
>Out[4]= 6927152
>Out[5]= 6654624
>
>Second, a situation which is not fine: I use the transposed array and memory usage explodes:
>
>In[1]:= MaxMemoryUsed[]
>MemoryInUse[]
>a=SparseArray[{{1,1}->1},{100000000,2}];
>MaxMemoryUsed[]
>MemoryInUse[]
>Out[1]= 5826816
>Out[2]= 6652184
>Out[4]= 806653720
>Out[5]= 406655144
>
>Thanks for any help.
>  
>



  • Prev by Date: Re: Product command with matrices
  • Next by Date: Re: Re: List concatenation - two more methods, one truly
  • Previous by thread: SparseArray memory usage
  • Next by thread: Re: SparseArray memory usage