Taking sums across indices of a SparseArray efficiently
- To: mathgroup at smc.vnet.net
- Subject: [mg95926] Taking sums across indices of a SparseArray efficiently
- From: "D. Grady" <D.C.Grady at gmail.com>
- Date: Fri, 30 Jan 2009 05:42:22 -0500 (EST)
Suppose we've got a four-dimensional array: t = Array[Subscript[w, ##] &, {3, 3, 3, 3}] If we want to take the sum across one index of this array (which will reduce its dimension), we can use the Total function: Dimensions@Total[t, {2}] {3, 3, 3} In the problem I'm working on, I've got an array and I need to sum across the first two dimensions. Using this toy array, I can see that Total[t,{1,2}] gives me exactly the object that I want. The problem is that I'm working with a four-dimensional sparse array, and Total will apparently always try to convert its first argument to a normal array. This fails because the array is too big to fit in memory: In[26]:= WAF = Total[W, {1, 2}]; // Timing During evaluation of In[26]:= SparseArray::ntb: Cannot convert the sparse array SparseArray[Automatic,{489,489,489,489},0,{<<1>>}] to an ordinary array because the 57178852641 elements required exceeds the current size limit. >> Out[26]= SystemException[SparseArrayNormalLimit,Normal[SparseArray [<1400152>,{489,489,489,489}]]] I can roll my own function to do this computation just by sorting through the ArrayRules: Timing[ WAF = SparseArray@( (#[[1, 1, 3 ;; 4]] -> Total[#[[All, 2]]] &) /@ (SplitBy[#, Drop[First@#, 2] &] &)@ (SortBy[#, Drop[First@#, 2] &] &)@ Most@ ArrayRules@ W)] {22.1335,SparseArray[<21122>,{489,489}]} The point is that actually doing the computation isn't particularly memory or time intensive, but I can't find a simple way to do this directly using built-in functions like Total. Does anyone know if there is a way? If there isn't, why not? Thanks a lot! -Daniel
- Follow-Ups:
- Re: Re: Taking sums across indices of a
- From: DrMajorBob <btreat1@austin.rr.com>
- Re: Taking sums across indices of a SparseArray efficiently
- From: DrMajorBob <btreat1@austin.rr.com>
- Re: Taking sums across indices of a SparseArray efficiently
- From: Carl Woll <carlw@wolfram.com>
- Re: Re: Taking sums across indices of a