MathGroup Archive 2011

[Date Index] [Thread Index] [Author Index]

Search the Archive

Sudden memory-usage increase with SparseArray

  • To: mathgroup at
  • Subject: [mg121150] Sudden memory-usage increase with SparseArray
  • From: Florian Heigl <floheigl at>
  • Date: Wed, 31 Aug 2011 06:06:07 -0400 (EDT)
  • Delivered-to:


I'm trying to build the transition Matrix for a pretty large (~700,000
vertices) graph. I'm doing this based on a list of almost 3 million edges.

When I tried to simple use something like:
 P=SparseArray[{#[[1]], #[[2]]} -> transProb & /@ Edges]
, i ran out of memory.

I've since increased the size of my swap partition, to give my 2+2 GB of
RAM+swap, which is - as far as I know - more than my kernel can address
anyway (32-bit linux).

When it still wouldn't work, I tried filling the array in steps of
10,000 entries, which is when I witnessed some strange behaviour:

Right up to around 70% of the entries to be added, memory only increases
slowly, so that it seems, I should be well within my system-constraints
for the whole thing, but then suddenly the process freezes, RAM gets
maxed out and swap soon after.

What's happening here?

P = SparseArray[{}, numstates, 0];
T0 = SparseArray[{}, numstates, 0];
step = 10000;

For[i = 1, i < numedges, i = i + step,
  T0 = SparseArray[Edges[[i ;; Min[i + step, Length[Edges]]]],
  P = P + T0;

  Florian Heigl

  • Prev by Date: Mathematica loop help
  • Next by Date: ODE system - shooting for a terminal point
  • Previous by thread: Mathematica loop help
  • Next by thread: Re: Sudden memory-usage increase with SparseArray