Re: Select, from table data

*To*: mathgroup at smc.vnet.net*Subject*: [mg112993] Re: Select, from table data*From*: "Nasser M. Abbasi" <nma at 12000.org>*Date*: Sun, 10 Oct 2010 06:41:20 -0400 (EDT)*References*: <i8pgih$fjn$1@smc.vnet.net>*Reply-to*: nma at 12000.org

On 10/9/2010 3:36 AM, Kurtis wrote: > Easy fix here. > > I'm trying to drop outliers from 2D data originally collected in the > time domain. My method is to analyze the 2nd derivative of smoothed > data and drop those deviating more than, say 3x the standard deviation > from the mean. I need to make a new table of the smoothed data with > these outliers dropped. I'm attempting the Select function in the > last line, i'm just not familiar enough with the syntax for analyzing > the criteria in the 2nd column (y values) from the dataset. Thanks! > > SetDirectory[NotebookDirectory[]]; > set = Import ["testMELT2.xls", {"Data", 1}]; > << Smooth` (* savitzsky-golay - like filter *) > > t = Table[set[[n]][[1]], {n, 1, Length[set]}]; I am a little confused (not too much xls exposure), but it seems to me that you have just imported one sheet (the first one), so set is just a matrix. So to get the first column, why not just write t=set[[All,1]] ? > a260 = Table[set[[n]][[2]], {n, 1, Length[set]}]; a260=set[[All,2]] ? > Dataset1 = Table[{t[[n]], a260[[n]]}, {n, 1, Length[set]}]; > Dataset1=t[[All,1;;2]] ? (I'll leave the rest for someone else to figure out for now :) > > SmoothDatasetDERIV = Smooth[Dataset1, 13, 3, 2]; (* external package > filtering over 13 points, polynomial order 3, f'') > meanSECderiv = Mean[SmoothDatasetDERIV]; > meanY = meanSECderiv[[2]] (* pulling out Y values *) > sd = StandardDeviation[SmoothDatasetDERIV]; > sdY = sd[[2]] (* pulling out Y values *) > > revision = > Select[{SmoothDatasetDERIV}, Function[Abs[meanY - #<= (3*sdY)&]]] > > > > Any help would be great, thanks! > > --Nasser