Re: Sudden memory-usage increase with SparseArray
- To: mathgroup at smc.vnet.net
- Subject: [mg121155] Re: Sudden memory-usage increase with SparseArray
- From: Oliver Ruebenkoenig <ruebenko at wolfram.com>
- Date: Wed, 31 Aug 2011 07:41:16 -0400 (EDT)
- Delivered-to: l-mathgroup@mail-archive0.wolfram.com
- References: <201108311006.GAA12167@smc.vnet.net>
On Wed, 31 Aug 2011, Florian Heigl wrote:
> Hi,
>
> I'm trying to build the transition Matrix for a pretty large (~700,000
> vertices) graph. I'm doing this based on a list of almost 3 million edges.
>
> When I tried to simple use something like:
> P=SparseArray[{#[[1]], #[[2]]} -> transProb & /@ Edges]
> , i ran out of memory.
Florian,
I think you could save some memory if you use something like this
SparseArray[ {{a1,b1},{a2,b2},..} -> {v1,v2,...}]
possibly as packed arrays.
Also, have a look if this helps:
g = Graph[{1 \[UndirectedEdge] 2, 2 \[UndirectedEdge] 3,
3 \[UndirectedEdge] 1}];
AdjacencyMatrix[g]
Hope this helps,
Oliver
>
> I've since increased the size of my swap partition, to give my 2+2 GB of
> RAM+swap, which is - as far as I know - more than my kernel can address
> anyway (32-bit linux).
>
> When it still wouldn't work, I tried filling the array in steps of
> 10,000 entries, which is when I witnessed some strange behaviour:
>
> Right up to around 70% of the entries to be added, memory only increases
> slowly, so that it seems, I should be well within my system-constraints
> for the whole thing, but then suddenly the process freezes, RAM gets
> maxed out and swap soon after.
>
> What's happening here?
>
> -------
> P = SparseArray[{}, numstates, 0];
> T0 = SparseArray[{}, numstates, 0];
> step = 10000;
>
> For[i = 1, i < numedges, i = i + step,
> T0 = SparseArray[Edges[[i ;; Min[i + step, Length[Edges]]]],
> numstates];
> P = P + T0;
> ];
> --------
>
> regards,
> Florian Heigl
>
>
- References:
- Sudden memory-usage increase with SparseArray
- From: Florian Heigl <floheigl@gmail.com>
- Sudden memory-usage increase with SparseArray