Re: Select, from table data
- To: mathgroup at smc.vnet.net
- Subject: [mg112993] Re: Select, from table data
- From: "Nasser M. Abbasi" <nma at 12000.org>
- Date: Sun, 10 Oct 2010 06:41:20 -0400 (EDT)
- References: <i8pgih$fjn$1@smc.vnet.net>
- Reply-to: nma at 12000.org
On 10/9/2010 3:36 AM, Kurtis wrote:
> Easy fix here.
>
> I'm trying to drop outliers from 2D data originally collected in the
> time domain. My method is to analyze the 2nd derivative of smoothed
> data and drop those deviating more than, say 3x the standard deviation
> from the mean. I need to make a new table of the smoothed data with
> these outliers dropped. I'm attempting the Select function in the
> last line, i'm just not familiar enough with the syntax for analyzing
> the criteria in the 2nd column (y values) from the dataset. Thanks!
>
> SetDirectory[NotebookDirectory[]];
> set = Import ["testMELT2.xls", {"Data", 1}];
> << Smooth` (* savitzsky-golay - like filter *)
>
> t = Table[set[[n]][[1]], {n, 1, Length[set]}];
I am a little confused (not too much xls exposure), but it seems to me
that you have just imported one sheet (the first one), so set is just a
matrix. So to get the first column, why not just write
t=set[[All,1]]
?
> a260 = Table[set[[n]][[2]], {n, 1, Length[set]}];
a260=set[[All,2]]
?
> Dataset1 = Table[{t[[n]], a260[[n]]}, {n, 1, Length[set]}];
>
Dataset1=t[[All,1;;2]]
?
(I'll leave the rest for someone else to figure out for now :)
>
> SmoothDatasetDERIV = Smooth[Dataset1, 13, 3, 2]; (* external package
> filtering over 13 points, polynomial order 3, f'')
> meanSECderiv = Mean[SmoothDatasetDERIV];
> meanY = meanSECderiv[[2]] (* pulling out Y values *)
> sd = StandardDeviation[SmoothDatasetDERIV];
> sdY = sd[[2]] (* pulling out Y values *)
>
> revision =
> Select[{SmoothDatasetDERIV}, Function[Abs[meanY - #<= (3*sdY)&]]]
>
>
>
> Any help would be great, thanks!
>
>
--Nasser