Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Re: Combining data from indexed lists efficiently

• To: mathgroup at smc.vnet.net
• Subject: [mg106253] Re: [mg106182] Combining data from indexed lists efficiently
• From: "Steve W. Brewer" <steve at take5.org>
• Date: Wed, 6 Jan 2010 06:00:26 -0500 (EST)
• References: <201001041059.FAA21250@smc.vnet.net> <4B42617D.2080204@wolfram.com>

Thanks everyone who responded!

I ended up going with Daniel's solution because it turned out to be easiest
to adapt to my specific situation.  However, I did need to make a couple of
changes to Daniel's code.

One of my derived-value functions (f2[]) uses the result of another
derived-value function (f1[]).  I could do something like this ...

    valueTables[ lists,
                 { f1[ #[[1]], #[[2]] ] &,
                   f2[ f1[ #[[1]], #[[2]] ], #[[3]] ] & }]

... but then f1[] is evaluated twice for each sample.  By changing

    vtj = Take[valtab[[j,2]],ll];

to

    vtj := valtab[[j,2]];

the complete result vector is available to my derived-value functions,
including results from previously computed functions.  So now I can write

    valueTables[ lists,
                 { f1[ #[[1]], #[[2]] ] &,
                   f2[ #[[4]], #[[3]] ] & }]

where #[[4]] is the result from f1.


I also discovered that some of my data had a few duplicate indices, which
didn't bother my original code but messed up the recordkeeping in the
subindices list in this approach.  I added a couple of lines of code to that
Do loop to skip over any duplicate indices encountered:

    Do[
        list = lists[[j]];
        previpos = 0;
        Do[
            listi = list[[i]];
            ipos = indexset[listi[[1]]];
            If [!IntegerQ[ipos] || ipos <= previpos, Continue[]];
            subindices[[ipos]] += 1;
            valtab[[ipos,2,subindices[[ipos]]]] = listi[[2]];
            previpos = ipos;
        ,{i,Length[list]}];
    ,{j,ll}];


Final result: My previous code took at least a couple of hours to run on my
machine.  The new code takes 19.5 seconds!


Here's the complete code with my changes:


indexesToUse[lists_List] := Apply[Intersection,Map[#[[All,1]]&,lists]]

initValueTable[indices_,len_] :=
   Table[{indices[[j]],ConstantArray[0,len]}, {j,Length[indices]}]

valueTables[lists_List,funcs_List] := Module[
   {indices,valtab,subindices,ll,list,listi,indexset,ipos,vtj},
   ll = Length[lists];
   indices = indexesToUse[lists];
   valtab = initValueTable[indices,ll+Length[funcs]];
   subindices = ConstantArray[0,Length[indices]];
   Do[indexset[indices[[j]]] = j, {j,Length[indices]}];
   Do[
     list = lists[[j]];
     previpos = 0;
	Do[
	  listi = list[[i]];
	  ipos = indexset[listi[[1]]];
	  If [!IntegerQ[ipos] || ipos <= previpos, Continue[]];
	  subindices[[ipos]] += 1;
	  valtab[[ipos,2,subindices[[ipos]]]] = listi[[2]];
	  previpos = ipos;
	  ,{i,Length[list]}];
     ,{j,ll}];
   Clear[indexset];
   Do[
	vtj := valtab[[j,2]];
     Do[
	  valtab[[j,2,ll+i]] = funcs[[i]][vtj];
	  ,{i,Length[funcs]}];
     ,{j,Length[valtab]}];
   valtab
   ]


Steve W. Brewer

• References:
  • Combining data from indexed lists efficiently
    • From: "Steve W. Brewer" <steve@take5.org>