Re: Sudden memory-usage increase with SparseArray
- To: mathgroup at smc.vnet.net
- Subject: [mg121155] Re: Sudden memory-usage increase with SparseArray
- From: Oliver Ruebenkoenig <ruebenko at wolfram.com>
- Date: Wed, 31 Aug 2011 07:41:16 -0400 (EDT)
- Delivered-to: l-mathgroup@mail-archive0.wolfram.com
- References: <201108311006.GAA12167@smc.vnet.net>
On Wed, 31 Aug 2011, Florian Heigl wrote: > Hi, > > I'm trying to build the transition Matrix for a pretty large (~700,000 > vertices) graph. I'm doing this based on a list of almost 3 million edges. > > When I tried to simple use something like: > P=SparseArray[{#[[1]], #[[2]]} -> transProb & /@ Edges] > , i ran out of memory. Florian, I think you could save some memory if you use something like this SparseArray[ {{a1,b1},{a2,b2},..} -> {v1,v2,...}] possibly as packed arrays. Also, have a look if this helps: g = Graph[{1 \[UndirectedEdge] 2, 2 \[UndirectedEdge] 3, 3 \[UndirectedEdge] 1}]; AdjacencyMatrix[g] Hope this helps, Oliver > > I've since increased the size of my swap partition, to give my 2+2 GB of > RAM+swap, which is - as far as I know - more than my kernel can address > anyway (32-bit linux). > > When it still wouldn't work, I tried filling the array in steps of > 10,000 entries, which is when I witnessed some strange behaviour: > > Right up to around 70% of the entries to be added, memory only increases > slowly, so that it seems, I should be well within my system-constraints > for the whole thing, but then suddenly the process freezes, RAM gets > maxed out and swap soon after. > > What's happening here? > > ------- > P = SparseArray[{}, numstates, 0]; > T0 = SparseArray[{}, numstates, 0]; > step = 10000; > > For[i = 1, i < numedges, i = i + step, > T0 = SparseArray[Edges[[i ;; Min[i + step, Length[Edges]]]], > numstates]; > P = P + T0; > ]; > -------- > > regards, > Florian Heigl > >
- References:
- Sudden memory-usage increase with SparseArray
- From: Florian Heigl <floheigl@gmail.com>
- Sudden memory-usage increase with SparseArray