Re: filtering data

*To*: mathgroup at smc.vnet.net*Subject*: [mg25365] Re: [mg25326] filtering data*From*: Daniel Lichtblau <danl at wolfram.com>*Date*: Sun, 24 Sep 2000 03:01:33 -0400 (EDT)*References*: <200009230735.DAA19278@smc.vnet.net>*Sender*: owner-wri-mathgroup at wolfram.com

steven.w.gaskey at exxon.com wrote: > > I am trying to apply a type of filter to a series of data. The filter is of the form > > y(i+1) = (1-p) y(i) + p/2 ( (1-p) x(i) + x(i+1)). > > where y is the filtered data series and x is the raw data series. > > First, I implemented this by defining the function for the single step then using FoldList to calculate the filtered series. > > BAStep[abai_, p_, {ai_, aip1_}] := (1 - p)abai + p/2((1 - p)ai + aip1) > > BA[a_, p_] := FoldList[BAStep[#1, p, #2] &, a[[1, 1]], a] > > Everything worked fine. However, this required me to partition the list first: > > ap = Partition[a,2,1]. > > When I moved the partition into the definition of the BA function: > > BA2[a_, p_] := FoldList[BAStep[#1, p, #2] &, First[a], Partition[a, 2, 1]] > > The application of the function got very slow and used a huge amount of memory. I applied this to two lists of 1500 points and it used all the system memory, crashed the front end and froze the kernel. > > My questions: > > 1. Is there a better way to implement a filter such as this. > > 2. What was the cause of the large use of memory and long time in applying the filter? > > Thanks for your help > > Steve > > Steven Gaskey > Staff Engineer, Exxpol Reactor Technology Development > ExxonMobil Chemical Company > Baytown Polymer Center > 5200 Bayway Dr > Baytown, TX 77520 > 281 834 0356 Fax: 281 834 2434 > Steven.W.Gaskey at Exxon.com Not sure I follow what you want to do, but maybe something like this? len = 10000; p = .6 xx = Table[Random[], {len}]; xx = Prepend[xx, First[xx]]; last = 0; Timing[yy = Table[ last = (1-p)*last + p/2*((1-p)*xx[[i]] + xx[[i+1]]); last, {i, 1, len}];] Takes about 1.5 seconds on my machine. Note that this relies on giving a specific numeric value for p, otherwise there will be significant symbolic swell. Even then it can be made more reasonable by use of interleaved Expand. After clearing p, I do: len = 1000; (* p = .6 *) xx = Table[Random[], {len}]; xx = Prepend[xx, First[xx]]; last = 0; Timing[yy = Table[ last = Expand[(1-p)*last + p/2*((1-p)*xx[[i]] + xx[[i+1]])]; last, {i, 1, len}];] and get Out[58]= {45.55 Second, Null} In[60]:= LeafCount[yy] Out[60]= 2506501 Daniel Lichtblau Wolfram Research

**References**:**filtering data***From:*steven.w.gaskey@exxon.com