MathGroup Archive 2010

[Date Index] [Thread Index] [Author Index]

Search the Archive

Select, from table data

  • To: mathgroup at
  • Subject: [mg112987] Select, from table data
  • From: Kurtis <djroue at>
  • Date: Sat, 9 Oct 2010 06:35:29 -0400 (EDT)

Easy fix here.

I'm trying to drop outliers from 2D data originally collected in the
time domain.  My method is to analyze the 2nd derivative of smoothed
data and drop those deviating more than, say 3x the standard deviation
from the mean. I need to make a new table of the smoothed data with
these outliers dropped.  I'm attempting the Select function in the
last line, i'm just not familiar enough with the syntax for analyzing
the criteria in the 2nd column (y values) from the dataset.  Thanks!

set = Import ["testMELT2.xls", {"Data", 1}];
<< Smooth` (* savitzsky-golay - like filter *)

t = Table[set[[n]][[1]], {n, 1, Length[set]}];
a260 = Table[set[[n]][[2]], {n, 1, Length[set]}];
Dataset1 = Table[{t[[n]], a260[[n]]}, {n, 1, Length[set]}];

SmoothDatasetDERIV = Smooth[Dataset1, 13, 3, 2]; (* external package
filtering over 13 points, polynomial order 3, f'')
meanSECderiv = Mean[SmoothDatasetDERIV];
meanY = meanSECderiv[[2]]  (* pulling out Y values *)
sd = StandardDeviation[SmoothDatasetDERIV];
sdY = sd[[2]]  (* pulling out Y values *)

revision =
 Select[{SmoothDatasetDERIV}, Function[Abs[meanY - # <= (3*sdY) &]]]

Any help would be great, thanks!

  • Prev by Date: Re: A list based table join?
  • Next by Date: Re: C-pointers from Mathematica
  • Previous by thread: GeneralizedLinearModelFit and offsert for poisson regression
  • Next by thread: Re: Select, from table data