Functionality and Reliability

*To*: mathgroup at smc.vnet.net*Subject*: [mg34958] Functionality and Reliability*From*: Name <mee at home.com.redline.ru>*Date*: Sat, 15 Jun 2002 02:27:45 -0400 (EDT)*Sender*: owner-wri-mathgroup at wolfram.com

As is the case with many other types of software, the functionality offered by competing systems is very much the same. The effort needed to achieve the result becomes the main difference. Functional differences such as faster and more robust numerical computations or better handling of improper integrals in Mathematica will probably be less important to the user than optimizations done in Mathematica for making the work more efficient: shorthand notation for lambda expressions, clear and compact syntax for pattern matching, shortcuts for 2D input. One of the major problems with Mathematica seems to be its lack of reliability. Some of us still remember "Out of memory. Exiting" of the third version, and it's not unusual to see Mathematica enter an infinite loop on some tricky sum or integral. But, in my opinion, the problem lies deeper than that. First of all, testing and quality assurance seem poor and inadequate. As an example, in the LaplaceTransform module there is a bug which is essentially just a simple syntax error. Namely, the Head function is being called with two arguments. This error existed in version 3.0...and it still exists in 4.1! If such a bug has been overlooked, it means that significant parts of the program were written but never tested, not even on so-called 'first cover' test sets. But it should be obvious that one of the major difficulties with testing this kind of software lies in the validation of the results. It follows that possible employment of interactive beta-testing in such cases is limited and creation of wide cover test suites becomes more important. What is equally important, some errors and cases of unexpected behaviour of the program should be attributed to sloppy design decisions: limit for integrals involving DiracDelta is sometimes taken to be -0 and sometimes +0 (this also stems from inaccurate design, namely taking integrals of DiracDelta on [0,x] without specifying what that means); in some cases Mathematica doesn't distinguish between UnitStep[x,y] and UnitStep[x y] or ArcTan[x,y] and ArcTan[y/x]. Some inconsistencies can be very annoying and not at all obvious; thus, there are examples where taking limit of x behaves differently from limit of (-x), or where setting PrincipalValue option for Integrate to True will prevent integrals convergent in the ordinary sense from being evaluated. Is it just some minor inconsistencies? Mathematica is very cavalier in its implementation of some things. Giving Taylor series expansion about a point on a branch cut? Sure. Returning finite results for sums diverging to infinity? Okay. Taking integrals across branch cuts? Why not. Trying to calculate integrals involving functions of the form Min[Sin[a x],Cos[b x]] by using some (very unreliable) heuristics? Sounds like fun. Moreover, even for integrating rational functions along straight lines in complex plane, where rigorous algorithms are possible at cost of returning more complicated expressions involving complex and algebraic numbers, the choice was made in favor of 'real-in, real-out' paradigm. This is a strange situation, because essentially it means preferring 'neat' answer to correct one! Some of the arguments in defense of this approach are: a) without simplifications like x/x=1 no symbolic computation will get anywhere; b) in many cases, such as antiderivatives with branch cuts, there is no general algorithm that is guaranteed to work; c) the goal is to make reasonable number of 'standard' examples work while acknowledging that more 'pathological' ones can probably fail. Besides, there are many design issues where the answer is by no means obvious. In many cases there appears a dilemma of whether to give an answer correct only for a certain domain of the input parameter values or simply leave the input unevaluated. What should be done if the convergence of a sum/product cannot be established explicitly? Basically, there are three possible approaches: giving a formal closed form expression without trying to check the convergence; generating answers in the form of conditional expressions depending on the parameters; leaving the sum unevaluated unless convergence can be positively established. Generating conditions seems to be the optimal way; besides, it would be consistent with Mathematica's handling of integrals. Mathematica usually takes the first approach, although in simpler cases, where Mathematica is able to establish the divergence explicitly, the sum is left unevaluated, so the motto is 'everything is correct that is not proven incorrect'. An important question is whether such 'simplifications' or, more precisely, unjustified assumptions can lead to evident errors, ie results that are manifestly incorrect for given inputs. Unfortunately, the answer is yes; in general, it is safe to assume that if we get an answer that is correct only for some values of the input parameters, then there will be examples where the result is incorrect for all inputs. Thus, if Limit[Sqrt[-1+a x],x->0] is correct for some values of a, Limit[Sqrt[-1-I Abs[a]x],x->0] is wrong for all values of a other than zero. If Limit[x f[x],x->0] gives zero, Limit[Log[a^x]/Log[a],a->0] will also give zero, which is not so good. Besides, this raises the question of inconsistencies between different modules; for example, Series[Sqrt[x],{x,-1,1}] and Limit[Sqrt[-1-I x],x->0] do not correlate, so double work has to be done implementing Limit, and nonetheless there will be examples (PolyLog) where Limit, using output from Series, will return incorrect results. Errors in Limit in turn can affect Integrate. So the most important consequence of the above is that other modules will be affected. Inconsistent handling of DiracDelta leads to errors in integral transform functions. Computing Taylor series where in fact it doesn't exist will lead to incorrect numerical evaluation of the function if such a series is used for that purpose, which is what indeed seems to happen with PolyLog at the branch point. Giving finite results for divergent sums (by analytic continuation) seems harmless in comparison, since it doesn't lead to complications in other modules...hopefully. The bottom line is that such an approach will lead to errors not only in some exotic Kolmogorov style examples and the errors will be quite hard to trace. From the developer's point of view, such problems probably have a low priority; the questions discussed here are probably viewed as something like a necessary evil. But from the user's point of view, this returns us to the issue of functionality and ease of use; the effort needed to achieve the result increases greatly if the user has to trace and overcome such difficulties. Besides, it is easy to predict that problems of this kind only get worse as the software gets more complicated. What suggestions can be offered here? First, to avoid 'partially correct' results whenever possible. Sum and Product should give conditional outputs just like Integrate. Perhaps assumptions mechanism should be extended to other functions such as Limit. One of the ways for handling (some) discontinuous antiderivatives is implementing singularity search functions. Inconsistencies across different modules should be removed. Another suggestion, no doubt useful for solving the problem of inconsistencies as well as for quality assurance, is submitting the program's specifications/design documents to an independent testing group; some authors suggest it for any large software project. Of course, those changes will come at a price. In certain cases Mathematica will not 'dare' give a result, being unable to establish its correctness. Some users will not be happy about getting overly complicated results in seemingly simple cases. Some of the problems considered above do not have a universal solution at all. Does it mean that computer algebra systems (or let's call them symbolic mathematics systems) are unreliable in general? [Please do not quote the whole message if you wish to reply] Maxim Rytin m.r at prontomail.com

**Re: Help: Why no output?**

**Definitions of the functions**

**RE: Re: RE: Experimental`FileBrowse**

**RE: Functionality and Reliability**