Re: Re: error with Sum and Infinity

*To*: mathgroup at smc.vnet.net*Subject*: [mg102443] Re: [mg102437] Re: error with Sum and Infinity*From*: Richard Fateman <fateman at cs.berkeley.edu>*Date*: Sun, 9 Aug 2009 18:19:26 -0400 (EDT)*References*: <h5bk64$hlm$1@smc.vnet.net> <200908070932.FAA15211@smc.vnet.net> <29EAD82D-4D06-4036-923E-97DD7B2EDC46@mimuw.edu.pl> <4A7C6176.9060002@cs.berkeley.edu> <h5jdik$18j$1@smc.vnet.net> <200908091006.GAA16013@smc.vnet.net> <99330D0E-4701-4243-BC10-B5A0E9242C03@mac.com>

George Woodrow III wrote: > Also including a private response from Richard: > > [me] >> You can get the performance you want by using Piecewise[]. That way, >> there is only one function definition. >> > [Richard] > Yes, one can often get the predetermined right answer by asking a > subtlely different question. Still, I think you agree the answer to > the original question was wrong... > > [my comment] > The point to this is that yes, the original 'naive' attempt at a sum > gave the wrong answer. The wrong answer should have prompted a journey > to find out why. Advising the user to read the manual is so common that there is a heavily-used acronym for it (RTFM). But in this case the manual probably does not provide enough guidance about what Sum does. > > The reason why the first attempt gave the wrong answer is that the > question was ill-posed for Mathematica to "always" do the right thing. You and I know that Mathematica didn't do the right thing. But there was no way for a user to determine if the problem was ill-posed in terms of Mathematica. Indeed, I can think of a technique that would allow Mathematica to solve such problems, and I have no reason to believe, a priori, that Mathematica does not have that technique. Furthermore, Mathematica may have such a technique in some future version. In particular, if Mathematica does not apparently intend to distinguish between a mathematical function composed of Sin Cos Pi + - * and a user-defined pattern-driven computer subroutine f[x_]:=If[ ...], then it should work regardless of how the expression is conveyed. The problem is WELL FORMED! > My solution (using Piecewise) and another poster's solution using > DiracDelta posed the question in a way that when Mathematica switched > from brute force (below SymbolicSumThreshold) to symbolic, it would do > the right thing. Another solution would be to do the problem in your head, and write down the answer, since it was obvious. This is making excuses for Mathematica, don't you think? <snip.. example> > The point of this is to show that you need to know some maths, and at > least a little bit about the conventions of Mathematica. An infinite > sum cannot be treated the same as a finite sum, and care needs to be > taken to be sure that Mathematica knows what the sequence really is. But in addition to knowing some maths, you also need to know things about Mathematica that it had no business expecting you to know. That's what I object to here. > > In the case of a sequence of all 0, except for a single 1, then the > function that defines the sequence needs to make the exception clear > in the definition. > > Could Mathematica be designed to always give the right answer, not > matter what the input? Perhaps. Most of the time, it would be a waste > of time. Should you make an excuse for Mathematica getting the wrong answer, even if it could, with some slight amount of additional cleverness get the right answer? No, I think you should make a fuss. Maybe just a slight amount of fuss, but some fuss. To show that you are paying attention. I make a fuss because I know that the program can be written properly. You think it would be a waste of time to write the program properly. That's a fundamental disagreement between us. I elaborate further below. > > The proper lesson to learn is that if you are going to be evaluating > infinite sums, then it would be worth finding out how Mathematica does > this and what needs to be done to make the code fool-proof. No, I think the lesson is that Mathematica documentation does not describe what it does with functions that are not "closed form" in infinite summations or even "large summations with SumThresholdWhatever terms", and that there is nothing you can do about such undocumented "features" except try to program around them (and in my case, also complain). Note that the default values or even the existence of SystemOptions can change without notice from version to version, and so you cannot possibly "find out" about such things all the time. It is simply a bad idea for Mathematica to behave this way, and for you to make the excuse that the user of Mathematica is responsible for making his code fool-proof in the light of this design is to place an enormous and largely unnecessary burden on someone who has paid good money for this program. I believe that Stephen Wolfram's perspective on this is that no one should believe the result of a computer calculation without independent verification, and therefore it is acceptable to take shortcuts which work most of the time. But not always. This is apparently OK for (some) physicists. It is generally unacceptable to computer scientists who do research in this area. It should be unacceptable to mathematicians, except perhaps those who were trained by physicists. > > In this case, one could learn that Mathematica will sometimes ignore > exceptional cases (separate definitions of the function at special > values) when computing sums, and if you want these cases to be > included in an infinite sum, you need to define the function so that > these cases are handled symbolically. You could learn it, but mostly by treating Mathematica, the computer program, as though it were a dead frog, and you were a curious child. You wonder if a frog, which lives in the water, breathes with gills or lungs. So you dissect it. The advantage of a computer program over a corpse is that you can get any number of fresh corpses to dissect, and so you can come to learn all kinds of things. Unfortunately, some of the things you learn may be incorrect generalizations from specific tests. I would prefer that Mathematica behave in a predictable, documented, and mathematically consistent way, and that I not be required to poke it and check that my programs still work when I install a new version, or when I run the program with different input parameters that might, unknown to me, exceed some threshold that I never heard of. > > You can be upset that mathematica does not do exactly what you expect > all the time, but it does no good to rant about it. It depends on your expectation being reasonable or not, don't you think? > The fact is, there are well documented ways to deal with this problem > so that mathematica *does* do the right thing, not just in this toy > example, but in more complex problems. Although I have hardly touched the vast array of contributions from users, I suspect that your claim is not true for this case or for some of the more complex problems. Where is it documented (much less "well documented") that sums above 1000000 [or whatever] are different from below that? > Every language has its way of interpreting code. Mathematica is > special in that there are a lot more ways to get the code 'almost' > right and usually a lot of different ways to get the code exactly > right. A similar coding error in c would simply give a compiler error > or crash. This is true, but Mathematica has some pretense to doing mathematics, and you are blaming the user for making a programming error when there is nothing wrong with the program. The error is that Mathematica is not correctly interpreting the code, which is both legal, perfectly clear, and unambiguous, by taking a shortcut that is incorrect. Think about what you are saying. If a C programmer used a compiler and found that it sometimes took a correct program and ran it incorrectly, would he say, oh, I can write a different program that gets the right answer? This is why I view Mathematica, from a computer science/ programming language perspective as, how shall we say,... disappointing. So what is the issue? 1. WRI does not define what its way of interpreting code really is, and reserves the right to change it, anyway. 2. The Mathematica user is dependent on the good will of the implementation programmers and, to some extent, luck, for his program to work. 3. While guided by mathematics generally, the computer system programmers deliberately take shortcuts that violate 100% correctness. You are, of course, free to try to use Mathematica to solve problems, and it may work just fine most of the time. But as a computer scientist who is familiar with mathematics generally, and quite familiar with computer algebra systems, I find it unfortunate that you (and others) make excuses for Mathematica getting incorrect answers, even when other computer programs might get them right, (and WRI may be convinced to fix the issues too) thinking that somehow the USER is at fault. > > As for your gratuitous comment about Andrzej's post, I think that > anyone familiar with any type of computer algebra system would be able > to guess, just in terms of practicality, that there is a threshold > from brute force to symbolic summation. Why can't Mathematica figure out that the summand is defined in a peculiar way and has to be handled carefully? Why does the user have to think about whether the summation is done by brute force or by closed formula? What if the user is building a package and does not know what the user of his package has done to define a program that is, eventually, used as a summand? The package-builder must do the analysis that WRI should have done in the first place, or else that package may fail mysteriously. This means the package writer must violate a precept of the WRI way, which is that a Mathematica user need not be concerned about the details of algorithms. Maybe you THINK you know about how to do summation, but do you know how Mathematica does it? Does your knowledge extend to do (say) polynomial factorization? Are you expected to know which classes of polynomials are hard to factor and therefore you should not ask Mathematica to factor them? > Otherwise, Mathematica could never compute an infinite sum. I have > been using Mathematica for 20 years, and I did not know the name for > the threshold, let alone its value, but for my purposes, I did not > need to know. However, I did know (and it was in the documentation) > that there is a point where one type of computation ends and another > begins. > > The function SystemOptions[] is defined and documented in version 7 > (It was new in version 6). > > If you execute SystemOptions["*"], you get an impressive list of all > the options and their current values. Most appear to be > self-explanatory, but a few look like more explanation is needed. It > also appears that if your code depends on the values for these > options, the code may be fragile at a minimum. This is true. In fact I may have used SystemOptions[] at some point in the distant past (1984?) but did not connect it to Sum since there is no indication in the documentation for Sum that it depends on this threshold option. So there is no way, other than poking at the corpse, for you to tell if your code is fragile. > > Note that the original problem had a resolution that did not involve > special knowledge of these variables. Sure, there are lots of resolutions, including, as I said, the excellent one of the user just figuring out the answer. Face it, Mathematica got the wrong answer to a problem that is entirely well-posed. Stop making excuses! > I'd say that knowing that these options exist belongs to advanced > users of Mathematica. It might be nice to have more complete > documentation for some of these options -- and Tech Support has > provided this information when I have needed it, but this knowledge > would not have solved the original question. That is because the original problem was well posed, and Mathematica should have solved it!. Can you not conceive of a programming solution to the original problem, or at least a solution that determined whether this was a case that might lead to an error? {It might require going slightly outside the usual Mathematica techniques -- I don't know offhand how to decompose a function definition to see how many rules it includes, and do a kind of meta-analysis of the patterns. } ...<snip> RJF

**References**:**Re: error with Sum and Infinity***From:*Richard Fateman <fateman@cs.berkeley.edu>

**Re: error with Sum and Infinity***From:*Richard Fateman <fateman@cs.berkeley.edu>