Re: Optimizing FindMinimum for speed using intermediate calculations.

*To*: mathgroup at christensen.cybernetics.net*Subject*: [mg1238] Re: Optimizing FindMinimum for speed using intermediate calculations.*From*: rubin at msu.edu (Paul A. Rubin)*Date*: Fri, 26 May 1995 05:10:03 -0400*Organization*: Michigan State University

In article <3pjl4l$7h0 at news0.cybernetics.net>, Bill Harbaugh <harbaugh at students.wisc.edu> wrote: ->Hello -> ->I am trying to maximize a likelihood function, using FindMinimum. This ->involves finding the parameters (x,y in my simplified example) that max ->(min, in my example) the sum of the values of a function applied to some ->data (ddist, in my example). -> ->My actual problem is very long and my question is about how to optimize ->my mma code for speed. -> ->This function involves some intermediate results (x1,y1 in my example) ->which vary across the x,y parameters but are constant across the d's. ->(In my actual problem, these intermediate results are found using NSolve, ->so I am specifying the functions using := rather than =). I am trying to ->only calculate these once for each iteration on x,y, and then apply the ->results to each of the d's in the list ddist. The code below shows how I ->do this without trying to optimize. -> -> ->ddist={1,2,3,4,5,6,7,8,9,10} ->x1:=x+1 ->y1:=y+2 ->l[d_]:=(x1*d)^2+(y1*d)^2 -> ->f=Apply[Plus, -> Map[l,ddist] -> ] -> ->FindMinimum[f,{x,{-1,1}},{y,{-1,1}}] -> -> -> -> ->This works, but it seems the following should be faster. -> ->Clear[l]; ->Clear[f]; ->l[d_]:=(x1*d)^2+(y1*d)^2 -> ->f={x1=x+1,y1=y+2,Apply[Plus, -> Map[l,ddist] -> ]} -> ->FindMinimum[f[[3]],{x,{-1,1}},{y,{-1,1}}] -> ->This also works, but timing results are inconclusive. I would appreciate ->suggestions on any other approaches that might speed this up. I am also ->asking mma support for help, and will forward their reply. -> ->Bill Harbaugh ->harbaugh at students.wisc.edu -> I don't see why the second version should be faster. In both cases, you compute an expression (f in the first case, f[[3]] in the second) that is quadratic in x and y, and feed that to FindMinimum. The only difference in the two calls to FindMinimum is that the second forces FindMinimum to extract the third entry in a list (f), which would slow it down a tad. As to the steps leading up to FindMinimum, I don't see any meaningful difference. I suspect, though, that your example above does not illustrate your actual problem. In both versions of the example, you end up with an objective function directly computable from x and y. It sounds as though, in your actual problem, there is an intermediate step (getting x1, y1 from x and y) that cannot be written in closed form. That being the case, the question is: where are your CPU cycles going? Is the time consuming part getting from {x, y} to {x1, y1}, or getting from {x1, y1} to f (and its derivatives)? Or is it the sheer number of iterations you're doing? Off the cuff, the only suggestion I can give you for speeding up your problem (and it's a shot in the dark) would be to start with a subset of the data, solve using just the subset (which should go faster), then add back some of the deleted observations and solve again, using the previous solution as the starting values for FindMinimum. That way, the early iterations (getting you into the general vicinity of the optimum) are done at lower computational cost. You could also start with a fairly sloppy precision goal and/or iteration limit, and get pickier as you add points (and, hopefully, get closer to the optimum). Unfortunately, FindMinimum does not give you a list of intermediate results for x and y. Steepest descent has been known to zigzag its way toward an optimum. If you knew that was happening, you could switch to a fancier optimization method (something based on conjugate gradients or quasi-Newton methods) and perhaps get faster convergence. Then again, you'd have to program it yourself. Paul ************************************************************************** * Paul A. Rubin Phone: (517) 432-3509 * * Department of Management Fax: (517) 432-1111 * * Eli Broad Graduate School of Management Net: RUBIN at MSU.EDU * * Michigan State University * * East Lansing, MI 48824-1122 (USA) * ************************************************************************** Mathematicians are like Frenchmen: whenever you say something to them, they translate it into their own language, and at once it is something entirely different. J. W. v. GOETHE