More memory-efficient inner product for large last dimension?

*To*: mathgroup at smc.vnet.net*Subject*: [mg106870] More memory-efficient inner product for large last dimension?*From*: Vince Virgilio <blueschi at gmail.com>*Date*: Mon, 25 Jan 2010 05:09:20 -0500 (EST)

Hello, I need to compute vector-vector, matrix-vector, and matrix-matrix inner products, for vectors and matrices whose elements are not scalars, but very large lists (~ 1.2M element each). I need Dot[] to ignore the last tensor index, but it has no parameter for this, like Outer's last "n_i " arguments. So I implemented my own. Unfortunately, the matrix-vector and matrix-matrix products consume excessive amounts of memory. The matrix-vector product peak memory footprint is ~ 800MB, for ~ 110MB total input, and the matrix-matrix product peaks at ~ 1.8 GB for ~ 180MB input. Apparently, memory overhead is ~ 8-10X. Here is a trace of system memory use (working set and its peak) for the above Mathematica evaluations. My system is Windows 7, 2.5 GHz Intel Core 2 Duo, 4GB RAM (Lenove R61 laptop). http://tinyurl.com/yfgwp26 (PDF ~ 180KB) Please find below my implementation of "dot", which ignores sublists below level 1 or 2 (depends on product type). Can it be made more efficient? Thank you, Vince Virgilio dot[l1_, l2_, 1] := l1*l2 // Total; dot[l1_, l2_, 2, n2_: 1] := ReleaseHold@Dot[Map[Hold, l1, {2}], Map[Hold, l2, {n2}]]; In[3]:= l1 = RandomReal[1., {3, 1200000}]; l2 = RandomReal[1., {3, 3, 1200000}]; In[5]:= ByteCount@l1 Out[5]= 28800128 In[6]:= ByteCount@l2 Out[6]= 86400132 (* Vector-Vector inner product *) In[7]:= l3 = dot[l1, l1, 1]; Dimensions@l3 Out[8]= {1200000} (* Matrix-Vector inner product *) l4 = dot[l2, l1, 2]; (* Memory use peaks @ ~ 800 MB *) In[8]:= Dimensions@l4 Out[8]= {3, 1200000} (* Matrix-Matrix inner product *) l5 = dot[l2, l2, 2, 2]; (* Memory use peaks @ ~ 1.8 GB! *) In[8]:= Dimensions@l5 Out[8]= {3, 3, 1200000}

**Follow-Ups**:**Re: More memory-efficient inner product for large last***From:*Leonid Shifrin <lshifr@gmail.com>

**Re: More memory-efficient inner product for large last***From:*"Vincent N. Virgilio" <virgilio@ieee.org>

**Re: More memory-efficient inner product for large last***From:*"Vincent N. Virgilio" <virgilio@ieee.org>

**Re: More memory-efficient inner product for large last***From:*"Vincent N. Virgilio" <virgilio@ieee.org>

**Re: More memory-efficient inner product for large last***From:*Leonid Shifrin <lshifr@gmail.com>

**Re: More memory-efficient inner product for large last***From:*Leonid Shifrin <lshifr@gmail.com>

**Re: More memory-efficient inner product for large last***From:*"Vincent N. Virgilio" <virgilio@ieee.org>