Re: Trying to quickly split lists at the point of maximum variance reduction

*To*: mathgroup at smc.vnet.net*Subject*: [mg127623] Re: Trying to quickly split lists at the point of maximum variance reduction*From*: Ray Koopman <koopman at sfu.ca>*Date*: Wed, 8 Aug 2012 21:34:00 -0400 (EDT)*Delivered-to*: l-mathgroup@mail-archive0.wolfram.com*Delivered-to*: l-mathgroup@wolfram.com*Delivered-to*: mathgroup-newout@smc.vnet.net*Delivered-to*: mathgroup-newsend@smc.vnet.net*References*: <jvn3om$am6$1@smc.vnet.net>

On Aug 5, 5:39 pm, Earl Mitchell <earl.j.mitch... at gmail.com> wrote: > Hi all, > > I've got a list of 42,000 lists of 233 variables (integers from 1 to 250), > each with a response in the last position (integer from 0 to 9). I want to > find the variable and split point in that variable at which the weighted > average variance in the response will be minimized. > > As a trivial example say the original variance in the response for the > whole set is 100. Imagine that there is some variable, say at position > 220, which has Tally'd values of {{0,40,000},{1,2000}} and, if we split the > corresponding responses along this break point the weighted average > variance of the two lists is 80, which happens to be the lowest possible > resulting variance for any single split, on any single variable. > > How can I find this point quickly? This description might be confusing - > let me know if something is not clear. Thanks ahead of time for the help! > > Mitch > > PS. Currently this job can be done in with this code: > > FindMaxVarianceReductionSplit[data_List] := > Module[{transdata, splitvarreductionpairs, withoutputs, testsplits, split, > endvar, maxvarreduction}, > transdata = Transpose[data]; > splitvarreductionpairs = With[{outputs = transdata[[-1]]}, > ParallelTable[ > With[{inputs = transdata[[i]], startvar = N@NewVariance[outputs]}, > withoutputs = Thread[{inputs, outputs}]; > testsplits = Union[inputs]; > Table[ > With[{splitval = testsplits[[j]]}, > split = GatherBy[withoutputs, #[[1]] > splitval &]; > endvar = > Total[(N@Length[#]*NewVariance[#[[All, -1]]] & /@ split)/ > Length[withoutputs]]; > {splitval, startvar - endvar} > ] > , > {j, Length[testsplits]}]], {i, Length[Most[transdata]]}] > ]; > > maxvarreduction = Max[Flatten[splitvarreductionpairs, 1][[All, -1]]]; > Position[splitvarreductionpairs, maxvarreduction] > > ] > > ... on my brand spanking new MBP it completes in just under 1,000 seconds > being parallelized. I need this to run much faster to have any practical > applications. > > Thanks again! I find your description confusing. Here's how I read it: You have a table whose dimensions are 42000 x 233, and a corresponding vector of 42000 responses. All the values in the table are integers in [1, 250]. All the responses are integers in [0,9]. You want to choose a column, say k, and a value, say x, such that if you split the responses into two groups according as the k'th element in the corresponding row of the table is <= x or > x, you minimize the sum of squared deviations of the responses from their respective group means. Please confirm or correct that interpretation.