       Re: Combining data from indexed lists efficiently

• To: mathgroup at smc.vnet.net
• Subject: [mg106253] Re: [mg106182] Combining data from indexed lists efficiently
• From: "Steve W. Brewer" <steve at take5.org>
• Date: Wed, 6 Jan 2010 06:00:26 -0500 (EST)
• References: <201001041059.FAA21250@smc.vnet.net> <4B42617D.2080204@wolfram.com>

```Thanks everyone who responded!

I ended up going with Daniel's solution because it turned out to be easiest
to adapt to my specific situation.  However, I did need to make a couple of
changes to Daniel's code.

One of my derived-value functions (f2[]) uses the result of another
derived-value function (f1[]).  I could do something like this ...

valueTables[ lists,
{ f1[ #[], #[] ] &,
f2[ f1[ #[], #[] ], #[] ] & }]

... but then f1[] is evaluated twice for each sample.  By changing

vtj = Take[valtab[[j,2]],ll];

to

vtj := valtab[[j,2]];

the complete result vector is available to my derived-value functions,
including results from previously computed functions.  So now I can write

valueTables[ lists,
{ f1[ #[], #[] ] &,
f2[ #[], #[] ] & }]

where #[] is the result from f1.

I also discovered that some of my data had a few duplicate indices, which
didn't bother my original code but messed up the recordkeeping in the
subindices list in this approach.  I added a couple of lines of code to that
Do loop to skip over any duplicate indices encountered:

Do[
list = lists[[j]];
previpos = 0;
Do[
listi = list[[i]];
ipos = indexset[listi[]];
If [!IntegerQ[ipos] || ipos <= previpos, Continue[]];
subindices[[ipos]] += 1;
valtab[[ipos,2,subindices[[ipos]]]] = listi[];
previpos = ipos;
,{i,Length[list]}];
,{j,ll}];

Final result: My previous code took at least a couple of hours to run on my
machine.  The new code takes 19.5 seconds!

Here's the complete code with my changes:

indexesToUse[lists_List] := Apply[Intersection,Map[#[[All,1]]&,lists]]

initValueTable[indices_,len_] :=
Table[{indices[[j]],ConstantArray[0,len]}, {j,Length[indices]}]

valueTables[lists_List,funcs_List] := Module[
{indices,valtab,subindices,ll,list,listi,indexset,ipos,vtj},
ll = Length[lists];
indices = indexesToUse[lists];
valtab = initValueTable[indices,ll+Length[funcs]];
subindices = ConstantArray[0,Length[indices]];
Do[indexset[indices[[j]]] = j, {j,Length[indices]}];
Do[
list = lists[[j]];
previpos = 0;
Do[
listi = list[[i]];
ipos = indexset[listi[]];
If [!IntegerQ[ipos] || ipos <= previpos, Continue[]];
subindices[[ipos]] += 1;
valtab[[ipos,2,subindices[[ipos]]]] = listi[];
previpos = ipos;
,{i,Length[list]}];
,{j,ll}];
Clear[indexset];
Do[
vtj := valtab[[j,2]];
Do[
valtab[[j,2,ll+i]] = funcs[[i]][vtj];
,{i,Length[funcs]}];
,{j,Length[valtab]}];
valtab
]

Steve W. Brewer

```

• Prev by Date: Re: Re: Re: algebraic numbers
• Next by Date: Re: Re: Re: algebraic numbers
• Previous by thread: Re: Combining data from indexed lists efficiently
• Next by thread: Re: Combining data from indexed lists efficiently