       Re: Taking sums across indices of a SparseArray efficiently

• To: mathgroup at smc.vnet.net
• Subject: [mg96053] Re: Taking sums across indices of a SparseArray efficiently
• From: "D. Grady" <D.C.Grady at gmail.com>
• Date: Tue, 3 Feb 2009 06:32:52 -0500 (EST)
• References: <200901301042.FAA06408@smc.vnet.net> <gm0q2n\$rle\$1@smc.vnet.net>

```On Jan 31, 12:11 am, Carl Woll <ca... at wolfram.com> wrote:
> D. Grady wrote:
> >Suppose we've got a four-dimensional array:
>
> >t = Array[Subscript[w, ##] &, {3, 3, 3, 3}]
>
> >If we want to take the sum across one index of this array (which will
> >reduce its dimension), we can use the Total function:
>
> >Dimensions@Total[t, {2}]
> >{3, 3, 3}
>
> >In the problem I'm working on, I've got an array and I need to sum
> >across the first two dimensions.  Using this toy array, I can see that
> >Total[t,{1,2}] gives me exactly the object that I want.  The problem
> >is that I'm working with a four-dimensional sparse array, and Total
> >will apparently always try to convert its first argument to a normal
> >array.  This fails because the array is too big to fit in memory:
>
> >In:= WAF = Total[W, {1, 2}]; // Timing
>
> >During evaluation of In:= SparseArray::ntb: Cannot convert the
> >sparse array SparseArray[Automatic,{489,489,489,489},0,{<<1>>}] to an
> >ordinary array because the 57178852641 elements required exceeds the
> >current size limit. >>
>
> >Out= SystemException[SparseArrayNormalLimit,Normal[SparseArray
> >[<1400152>,{489,489,489,489}]]]
>
> >I can roll my own function to do this computation just by sorting
> >through the ArrayRules:
>
> >Timing[
> > WAF =
> >  SparseArray@(
> >    (#[[1, 1, 3 ;; 4]] -> Total[#[[All, 2]]] &) /@
> >     (SplitBy[#, Drop[First@#, 2] &] &)@
> >      (SortBy[#, Drop[First@#, 2] &] &)@
> >       Most@
> >        ArrayRules@
> >         W)]
>
> >{22.1335,SparseArray[<21122>,{489,489}]}
>
> >The point is that actually doing the computation isn't particularly
> >memory or time intensive, but I can't find a simple way to do this
> >directly using built-in functions like Total.  Does anyone know if
> >there is a way?  If there isn't, why not?  Thanks a lot!
>
> >-Daniel
>
> Another workaround is to use Dot. Here is a toy array:
>
> In:= toy =
>  SparseArray[{i_, j_, k_, l_} -> i + j - k + l, {4, 4, 4, 4}]
>
> Out= SparseArray[<252>,{4,4,4,4}]
>
> In:= Total[toy, {1, 2}]
>
> Out= {{80, 96, 112, 128}, {64, 80, 96, 112}, {48, 64, 80,
>   96}, {32, 48, 64, 80}}
>
> In:= SparseArray[_ -> 1, 4].(SparseArray[_ -> 1, 4].toy)
>
> Out= SparseArray[<16>,{4,4}]
>
> In:= % // Normal
>
> Out= {{80, 96, 112, 128}, {64, 80, 96, 112}, {48, 64, 80,
>   96}, {32, 48, 64, 80}}
>
> Carl Woll
> Wolfram Research

What if I want to sum across just the 3rd index of an n-dimensional
array?  Can this still be made to work?

-Daniel

```

• Prev by Date: Re: FourierTransform
• Next by Date: Re: MCMC in Mathematica
• Previous by thread: FW: Split Screen
• Next by thread: simplifying a resulted derivative expression