       Re: Filtering data from numerical minimization

• To: mathgroup at smc.vnet.net
• Subject: [mg116250] Re: Filtering data from numerical minimization
• From: Ray Koopman <koopman at sfu.ca>
• Date: Tue, 8 Feb 2011 05:07:03 -0500 (EST)
• References: <iig75a\$rkb\$1@smc.vnet.net>

```On Feb 3, 10:42 pm, Sebastian <sebho... at gmail.com> wrote:
> I have a function f(n,x,y,z) that I want to numerically minimize with
> respect to x,y,z for a list of n=1...N. The function is pretty
> complicated and also contains other parameters, which are fixed as far
> as optimization is concerned. I use NMinimize to create a table for
> different values of n. This effectively leaves me with 4 lists (f_min,
> x_min,...), where f_min = f(x_min, y_min, z_min). I then plot these
> lists against the vector n.
>
> Depending on the other parameters the minimization sometimes works
> well, and sometimes it doesn't. The problem is the following: while
> the overall form of the curves is often easily recognizable by eye,
> there is some noise on top of it, i.e. some of the points (every 5th
> say) are just way off. As I don't think that f behaves that way, but
> rather NMinimize fails to find the correct value, I'd like to filter
> that noise. Filtering the points by hand, however, doesn't seem like a
> feasible solution. Does anyone know a good way to tackle this problem?
> I already tried running MovingMedian which works reasonably well but

My previous post ignored the next line:

> I'd like to keep the rest of the data untouched if possible.

It sounds like you're doing nonlinear regression and you have a fair
number of dependent-variable outliers, on the order of 20%. The usual
approach to such situations is to do some form of robust regression,
that minimizes something other than the unweighted sum of squared
residuals. There are many possibilities. Can you be more specific