Re: Sudden memory-usage increase with SparseArray
- To: mathgroup at smc.vnet.net
- Subject: [mg121151] Re: Sudden memory-usage increase with SparseArray
- From: Florian Heigl <floheigl at gmail.com>
- Date: Wed, 31 Aug 2011 06:06:18 -0400 (EDT)
- Delivered-to: l-mathgroup@mail-archive0.wolfram.com
- References: <4E5CC1A9.40307@gmail.com>
Hello again,
I'm terribly sorry to have bothered you - it turned out, my data was
corrupted. Because of a previous bug in my code, my list "Edges" had
some elements of the form {a,{b}}->val, instead of {a,b}->val. Those
elements occured far back in the list, hence the weird behaviour at that
point.
Now I'm stuck at trying to get NullSpace[] to handle an Array of the
form: SparseArray[<3686489>,{737298,737298}], which I guess is a whole
different story (and might ultimately be impossible, hints welcome
though...)
regards,
Florian Heigl
On 2011-08-30 12:55, Florian Heigl wrote:
> Hi,
>
> I'm trying to build the transition Matrix for a pretty large (~700,000
> vertices) graph. I'm doing this based on a list of almost 3 million edges.
>
> When I tried to simple use something like:
> P=SparseArray[{#[[1]], #[[2]]} -> transProb & /@ Edges]
> , i ran out of memory.
>
> I've since increased the size of my swap partition, to give my 2+2 GB of
> RAM+swap, which is - as far as I know - more than my kernel can address
> anyway (32-bit linux).
>
> When it still wouldn't work, I tried filling the array in steps of
> 10,000 entries, which is when I witnessed some strange behaviour:
>
> Right up to around 70% of the entries to be added, memory only increases
> slowly, so that it seems, I should be well within my system-constraints
> for the whole thing, but then suddenly the process freezes, RAM gets
> maxed out and swap soon after.
>
> What's happening here?
>
> -------
> P = SparseArray[{}, numstates, 0];
> T0 = SparseArray[{}, numstates, 0];
> step = 10000;
>
> For[i = 1, i < numedges, i = i + step,
> T0 = SparseArray[Edges[[i ;; Min[i + step, Length[Edges]]]],
> numstates];
> P = P + T0;
> ];
> --------
>
> regards,
> Florian Heigl