MathGroup Archive 2011

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Sudden memory-usage increase with SparseArray

  • To: mathgroup at smc.vnet.net
  • Subject: [mg121155] Re: Sudden memory-usage increase with SparseArray
  • From: Oliver Ruebenkoenig <ruebenko at wolfram.com>
  • Date: Wed, 31 Aug 2011 07:41:16 -0400 (EDT)
  • Delivered-to: l-mathgroup@mail-archive0.wolfram.com
  • References: <201108311006.GAA12167@smc.vnet.net>

On Wed, 31 Aug 2011, Florian Heigl wrote:

> Hi,
>
> I'm trying to build the transition Matrix for a pretty large (~700,000
> vertices) graph. I'm doing this based on a list of almost 3 million edges.
>
> When I tried to simple use something like:
> P=SparseArray[{#[[1]], #[[2]]} -> transProb & /@ Edges]
> , i ran out of memory.

Florian,

I think you could save some memory if you use something like this

SparseArray[ {{a1,b1},{a2,b2},..} -> {v1,v2,...}]

possibly as packed arrays.

Also, have a look if this helps:

g = Graph[{1 \[UndirectedEdge] 2, 2 \[UndirectedEdge] 3,
     3 \[UndirectedEdge] 1}];

AdjacencyMatrix[g]

Hope this helps,
Oliver

>
> I've since increased the size of my swap partition, to give my 2+2 GB of
> RAM+swap, which is - as far as I know - more than my kernel can address
> anyway (32-bit linux).
>
> When it still wouldn't work, I tried filling the array in steps of
> 10,000 entries, which is when I witnessed some strange behaviour:
>
> Right up to around 70% of the entries to be added, memory only increases
> slowly, so that it seems, I should be well within my system-constraints
> for the whole thing, but then suddenly the process freezes, RAM gets
> maxed out and swap soon after.
>
> What's happening here?
>
> -------
> P = SparseArray[{}, numstates, 0];
> T0 = SparseArray[{}, numstates, 0];
> step = 10000;
>
> For[i = 1, i < numedges, i = i + step,
>  T0 = SparseArray[Edges[[i ;; Min[i + step, Length[Edges]]]],
>    numstates];
>  P = P + T0;
>  ];
> --------
>
> regards,
>  Florian Heigl
>
>




  • Prev by Date: Re: FullSimplify and Coefficient
  • Next by Date: Re: Problem with access nested list
  • Previous by thread: Sudden memory-usage increase with SparseArray
  • Next by thread: Re: Sudden memory-usage increase with SparseArray