Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Sudden memory-usage increase with SparseArray

• To: mathgroup at smc.vnet.net
• Subject: [mg121150] Sudden memory-usage increase with SparseArray
• From: Florian Heigl <floheigl at gmail.com>
• Date: Wed, 31 Aug 2011 06:06:07 -0400 (EDT)
• Delivered-to: l-mathgroup@mail-archive0.wolfram.com

Hi,

I'm trying to build the transition Matrix for a pretty large (~700,000
vertices) graph. I'm doing this based on a list of almost 3 million edges.

When I tried to simple use something like:
 P=SparseArray[{#[[1]], #[[2]]} -> transProb & /@ Edges]
, i ran out of memory.

I've since increased the size of my swap partition, to give my 2+2 GB of
RAM+swap, which is - as far as I know - more than my kernel can address
anyway (32-bit linux).

When it still wouldn't work, I tried filling the array in steps of
10,000 entries, which is when I witnessed some strange behaviour:

Right up to around 70% of the entries to be added, memory only increases
slowly, so that it seems, I should be well within my system-constraints
for the whole thing, but then suddenly the process freezes, RAM gets
maxed out and swap soon after.

What's happening here?

-------
P = SparseArray[{}, numstates, 0];
T0 = SparseArray[{}, numstates, 0];
step = 10000;

For[i = 1, i < numedges, i = i + step,
  T0 = SparseArray[Edges[[i ;; Min[i + step, Length[Edges]]]],
    numstates];
  P = P + T0;
  ];
--------

regards,
  Florian Heigl

• Follow-Ups:
  • Re: Sudden memory-usage increase with SparseArray
    • From: Oliver Ruebenkoenig <ruebenko@wolfram.com>