Select, from table data
- To: mathgroup at smc.vnet.net
- Subject: [mg112987] Select, from table data
- From: Kurtis <djroue at gmail.com>
- Date: Sat, 9 Oct 2010 06:35:29 -0400 (EDT)
Easy fix here.
I'm trying to drop outliers from 2D data originally collected in the
time domain. My method is to analyze the 2nd derivative of smoothed
data and drop those deviating more than, say 3x the standard deviation
from the mean. I need to make a new table of the smoothed data with
these outliers dropped. I'm attempting the Select function in the
last line, i'm just not familiar enough with the syntax for analyzing
the criteria in the 2nd column (y values) from the dataset. Thanks!
SetDirectory[NotebookDirectory[]];
set = Import ["testMELT2.xls", {"Data", 1}];
<< Smooth` (* savitzsky-golay - like filter *)
t = Table[set[[n]][[1]], {n, 1, Length[set]}];
a260 = Table[set[[n]][[2]], {n, 1, Length[set]}];
Dataset1 = Table[{t[[n]], a260[[n]]}, {n, 1, Length[set]}];
SmoothDatasetDERIV = Smooth[Dataset1, 13, 3, 2]; (* external package
filtering over 13 points, polynomial order 3, f'')
meanSECderiv = Mean[SmoothDatasetDERIV];
meanY = meanSECderiv[[2]] (* pulling out Y values *)
sd = StandardDeviation[SmoothDatasetDERIV];
sdY = sd[[2]] (* pulling out Y values *)
revision =
Select[{SmoothDatasetDERIV}, Function[Abs[meanY - # <= (3*sdY) &]]]
Any help would be great, thanks!