Re: Combining data from indexed lists efficiently

*To*: mathgroup at smc.vnet.net*Subject*: [mg106253] Re: [mg106182] Combining data from indexed lists efficiently*From*: "Steve W. Brewer" <steve at take5.org>*Date*: Wed, 6 Jan 2010 06:00:26 -0500 (EST)*References*: <201001041059.FAA21250@smc.vnet.net> <4B42617D.2080204@wolfram.com>

Thanks everyone who responded! I ended up going with Daniel's solution because it turned out to be easiest to adapt to my specific situation. However, I did need to make a couple of changes to Daniel's code. One of my derived-value functions (f2[]) uses the result of another derived-value function (f1[]). I could do something like this ... valueTables[ lists, { f1[ #[[1]], #[[2]] ] &, f2[ f1[ #[[1]], #[[2]] ], #[[3]] ] & }] ... but then f1[] is evaluated twice for each sample. By changing vtj = Take[valtab[[j,2]],ll]; to vtj := valtab[[j,2]]; the complete result vector is available to my derived-value functions, including results from previously computed functions. So now I can write valueTables[ lists, { f1[ #[[1]], #[[2]] ] &, f2[ #[[4]], #[[3]] ] & }] where #[[4]] is the result from f1. I also discovered that some of my data had a few duplicate indices, which didn't bother my original code but messed up the recordkeeping in the subindices list in this approach. I added a couple of lines of code to that Do loop to skip over any duplicate indices encountered: Do[ list = lists[[j]]; previpos = 0; Do[ listi = list[[i]]; ipos = indexset[listi[[1]]]; If [!IntegerQ[ipos] || ipos <= previpos, Continue[]]; subindices[[ipos]] += 1; valtab[[ipos,2,subindices[[ipos]]]] = listi[[2]]; previpos = ipos; ,{i,Length[list]}]; ,{j,ll}]; Final result: My previous code took at least a couple of hours to run on my machine. The new code takes 19.5 seconds! Here's the complete code with my changes: indexesToUse[lists_List] := Apply[Intersection,Map[#[[All,1]]&,lists]] initValueTable[indices_,len_] := Table[{indices[[j]],ConstantArray[0,len]}, {j,Length[indices]}] valueTables[lists_List,funcs_List] := Module[ {indices,valtab,subindices,ll,list,listi,indexset,ipos,vtj}, ll = Length[lists]; indices = indexesToUse[lists]; valtab = initValueTable[indices,ll+Length[funcs]]; subindices = ConstantArray[0,Length[indices]]; Do[indexset[indices[[j]]] = j, {j,Length[indices]}]; Do[ list = lists[[j]]; previpos = 0; Do[ listi = list[[i]]; ipos = indexset[listi[[1]]]; If [!IntegerQ[ipos] || ipos <= previpos, Continue[]]; subindices[[ipos]] += 1; valtab[[ipos,2,subindices[[ipos]]]] = listi[[2]]; previpos = ipos; ,{i,Length[list]}]; ,{j,ll}]; Clear[indexset]; Do[ vtj := valtab[[j,2]]; Do[ valtab[[j,2,ll+i]] = funcs[[i]][vtj]; ,{i,Length[funcs]}]; ,{j,Length[valtab]}]; valtab ] Steve W. Brewer

**References**:**Combining data from indexed lists efficiently***From:*"Steve W. Brewer" <steve@take5.org>