Re: SparseArray memory usage
- To: mathgroup at smc.vnet.net
- Subject: [mg87869] Re: [mg87839] SparseArray memory usage
- From: Carl Woll <carlw at wolfram.com>
- Date: Sat, 19 Apr 2008 03:32:51 -0400 (EDT)
- References: <200804180641.CAA12681@smc.vnet.net>
wpb wrote:
>Hi,
>
>An obvious advantage of SparseArray is that less memory is needed for sparse arrays. Unfortunately, this doesn't always seem to be the case, as I outline below sometimes almost as much memory is used as for an ordinary array.
>
>Can someone explain this, and hopefully tell me how to get around it?
>
>
It's because of the way SparseArrays are internally structured. It
stores a cumulative count of the number of nonzero elements in each row.
For a matrix with 10^8 rows, this information takes:
In[1]:= ByteCount[ConstantArray[1, 10^8]]
Out[1]= 400000076
This is the memory usage you are seeing.
Obviously, if you can structure your problem so that it has fewer rows,
your memory usage will go down.
Carl Woll
Wolfram Research
>First, a situation which is fine:
>Starting with a clean kernel, I get the following memory usage for a 2 x 100million sparse array:
>
>In[1]:= MaxMemoryUsed[]
>MemoryInUse[]
>a=SparseArray[{{1,1}->1},{2,100000000}];
>MaxMemoryUsed[]
>MemoryInUse[]
>
>Out[1]= 5826800
>Out[2]= 6652184
>Out[4]= 6927152
>Out[5]= 6654624
>
>Second, a situation which is not fine: I use the transposed array and memory usage explodes:
>
>In[1]:= MaxMemoryUsed[]
>MemoryInUse[]
>a=SparseArray[{{1,1}->1},{100000000,2}];
>MaxMemoryUsed[]
>MemoryInUse[]
>Out[1]= 5826816
>Out[2]= 6652184
>Out[4]= 806653720
>Out[5]= 406655144
>
>Thanks for any help.
>
>
- References:
- SparseArray memory usage
- From: wpb <w.p.bergsma@lse.ac.uk>
- SparseArray memory usage