Re: filtering data
- To: mathgroup at smc.vnet.net
- Subject: [mg25365] Re: [mg25326] filtering data
- From: Daniel Lichtblau <danl at wolfram.com>
- Date: Sun, 24 Sep 2000 03:01:33 -0400 (EDT)
- References: <200009230735.DAA19278@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
steven.w.gaskey at exxon.com wrote:
>
> I am trying to apply a type of filter to a series of data. The filter is of the form
>
> y(i+1) = (1-p) y(i) + p/2 ( (1-p) x(i) + x(i+1)).
>
> where y is the filtered data series and x is the raw data series.
>
> First, I implemented this by defining the function for the single step then using FoldList to calculate the filtered series.
>
> BAStep[abai_, p_, {ai_, aip1_}] := (1 - p)abai + p/2((1 - p)ai + aip1)
>
> BA[a_, p_] := FoldList[BAStep[#1, p, #2] &, a[[1, 1]], a]
>
> Everything worked fine. However, this required me to partition the list first:
>
> ap = Partition[a,2,1].
>
> When I moved the partition into the definition of the BA function:
>
> BA2[a_, p_] := FoldList[BAStep[#1, p, #2] &, First[a], Partition[a, 2, 1]]
>
> The application of the function got very slow and used a huge amount of memory. I applied this to two lists of 1500 points and it used all the system memory, crashed the front end and froze the kernel.
>
> My questions:
>
> 1. Is there a better way to implement a filter such as this.
>
> 2. What was the cause of the large use of memory and long time in applying the filter?
>
> Thanks for your help
>
> Steve
>
> Steven Gaskey
> Staff Engineer, Exxpol Reactor Technology Development
> ExxonMobil Chemical Company
> Baytown Polymer Center
> 5200 Bayway Dr
> Baytown, TX 77520
> 281 834 0356 Fax: 281 834 2434
> Steven.W.Gaskey at Exxon.com
Not sure I follow what you want to do, but maybe something like this?
len = 10000;
p = .6
xx = Table[Random[], {len}];
xx = Prepend[xx, First[xx]];
last = 0;
Timing[yy = Table[
last = (1-p)*last + p/2*((1-p)*xx[[i]] + xx[[i+1]]);
last,
{i, 1, len}];]
Takes about 1.5 seconds on my machine. Note that this relies on giving a
specific numeric value for p, otherwise there will be significant
symbolic swell. Even then it can be made more reasonable by use of
interleaved Expand. After clearing p, I do:
len = 1000;
(* p = .6 *)
xx = Table[Random[], {len}];
xx = Prepend[xx, First[xx]];
last = 0;
Timing[yy = Table[
last = Expand[(1-p)*last + p/2*((1-p)*xx[[i]] + xx[[i+1]])];
last,
{i, 1, len}];]
and get
Out[58]= {45.55 Second, Null}
In[60]:= LeafCount[yy]
Out[60]= 2506501
Daniel Lichtblau
Wolfram Research
- References:
- filtering data
- From: steven.w.gaskey@exxon.com
- filtering data