Functionality and Reliability
- To: mathgroup at smc.vnet.net
- Subject: [mg34958] Functionality and Reliability
- From: Name <mee at home.com.redline.ru>
- Date: Sat, 15 Jun 2002 02:27:45 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
As is the case with many other types of software, the functionality
offered by competing systems is very much the same. The effort needed to
achieve the result becomes the main difference. Functional differences
such as faster and more robust numerical computations or better handling
of improper integrals in Mathematica will probably be less important to
the user than optimizations done in Mathematica for making the work more
efficient: shorthand notation for lambda expressions, clear and compact
syntax for pattern matching, shortcuts for 2D input.
One of the major problems with Mathematica seems to be its lack of
reliability. Some of us still remember "Out of memory. Exiting" of the
third version, and it's not unusual to see Mathematica enter an infinite
loop on some tricky sum or integral. But, in my opinion, the problem
lies deeper than that.
First of all, testing and quality assurance seem poor and inadequate. As
an example, in the LaplaceTransform module there is a bug which is
essentially just a simple syntax error. Namely, the Head function is
being called with two arguments. This error existed in version 3.0...and
it still exists in 4.1! If such a bug has been overlooked, it means that
significant parts of the program were written but never tested, not even
on so-called 'first cover' test sets. But it should be obvious that one
of the major difficulties with testing this kind of software lies in the
validation of the results. It follows that possible employment of
interactive beta-testing in such cases is limited and creation of wide
cover test suites becomes more important.
What is equally important, some errors and cases of unexpected behaviour
of the program should be attributed to sloppy design decisions: limit
for integrals involving DiracDelta is sometimes taken to be -0 and
sometimes +0 (this also stems from inaccurate design, namely taking
integrals of DiracDelta on [0,x] without specifying what that means); in
some cases Mathematica doesn't distinguish between UnitStep[x,y] and
UnitStep[x y] or ArcTan[x,y] and ArcTan[y/x]. Some inconsistencies can
be very annoying and not at all obvious; thus, there are examples where
taking limit of x behaves differently from limit of (-x), or where
setting PrincipalValue option for Integrate to True will prevent
integrals convergent in the ordinary sense from being evaluated.
Is it just some minor inconsistencies? Mathematica is very cavalier in
its implementation of some things. Giving Taylor series expansion about
a point on a branch cut? Sure. Returning finite results for sums
diverging to infinity? Okay. Taking integrals across branch cuts? Why
not. Trying to calculate integrals involving functions of the form
Min[Sin[a x],Cos[b x]] by using some (very unreliable) heuristics?
Sounds like fun.
Moreover, even for integrating rational functions along straight lines
in complex plane, where rigorous algorithms are possible at cost of
returning more complicated expressions involving complex and algebraic
numbers, the choice was made in favor of 'real-in, real-out' paradigm.
This is a strange situation, because essentially it means preferring
'neat' answer to correct one!
Some of the arguments in defense of this approach are: a) without
simplifications like x/x=1 no symbolic computation will get anywhere; b)
in many cases, such as antiderivatives with branch cuts, there is no
general algorithm that is guaranteed to work; c) the goal is to make
reasonable number of 'standard' examples work while acknowledging that
more 'pathological' ones can probably fail.
Besides, there are many design issues where the answer is by no means
obvious. In many cases there appears a dilemma of whether to give an
answer correct only for a certain domain of the input parameter values
or simply leave the input unevaluated. What should be done if the
convergence of a sum/product cannot be established explicitly?
Basically, there are three possible approaches: giving a formal closed
form expression without trying to check the convergence; generating
answers in the form of conditional expressions depending on the
parameters; leaving the sum unevaluated unless convergence can be
positively established. Generating conditions seems to be the optimal
way; besides, it would be consistent with Mathematica's handling of
integrals. Mathematica usually takes the first approach, although in
simpler cases, where Mathematica is able to establish the divergence
explicitly, the sum is left unevaluated, so the motto is 'everything is
correct that is not proven incorrect'.
An important question is whether such 'simplifications' or, more
precisely, unjustified assumptions can lead to evident errors, ie
results that are manifestly incorrect for given inputs. Unfortunately,
the answer is yes; in general, it is safe to assume that if we get an
answer that is correct only for some values of the input parameters,
then there will be examples where the result is incorrect for all
inputs. Thus, if Limit[Sqrt[-1+a x],x->0] is correct for some values of
a, Limit[Sqrt[-1-I Abs[a]x],x->0] is wrong for all values of a other
than zero. If Limit[x f[x],x->0] gives zero, Limit[Log[a^x]/Log[a],a->0]
will also give zero, which is not so good.
Besides, this raises the question of inconsistencies between different
modules; for example, Series[Sqrt[x],{x,-1,1}] and Limit[Sqrt[-1-I
x],x->0] do not correlate, so double work has to be done implementing
Limit, and nonetheless there will be examples (PolyLog) where Limit,
using output from Series, will return incorrect results. Errors in Limit
in turn can affect Integrate.
So the most important consequence of the above is that other modules
will be affected. Inconsistent handling of DiracDelta leads to errors in
integral transform functions. Computing Taylor series where in fact it
doesn't exist will lead to incorrect numerical evaluation of the
function if such a series is used for that purpose, which is what indeed
seems to happen with PolyLog at the branch point. Giving finite results
for divergent sums (by analytic continuation) seems harmless in
comparison, since it doesn't lead to complications in other
modules...hopefully.
The bottom line is that such an approach will lead to errors not only in
some exotic Kolmogorov style examples and the errors will be quite hard
to trace. From the developer's point of view, such problems probably
have a low priority; the questions discussed here are probably viewed as
something like a necessary evil. But from the user's point of view, this
returns us to the issue of functionality and ease of use; the effort
needed to achieve the result increases greatly if the user has to trace
and overcome such difficulties. Besides, it is easy to predict that
problems of this kind only get worse as the software gets more
complicated.
What suggestions can be offered here? First, to avoid 'partially
correct' results whenever possible. Sum and Product should give
conditional outputs just like Integrate. Perhaps assumptions mechanism
should be extended to other functions such as Limit. One of the ways for
handling (some) discontinuous antiderivatives is implementing
singularity search functions. Inconsistencies across different modules
should be removed. Another suggestion, no doubt useful for solving the
problem of inconsistencies as well as for quality assurance, is
submitting the program's specifications/design documents to an
independent testing group; some authors suggest it for any large
software project.
Of course, those changes will come at a price. In certain cases
Mathematica will not 'dare' give a result, being unable to establish its
correctness. Some users will not be happy about getting overly
complicated results in seemingly simple cases. Some of the problems
considered above do not have a universal solution at all. Does it mean
that computer algebra systems (or let's call them symbolic mathematics
systems) are unreliable in general?
[Please do not quote the whole message if you wish to reply]
Maxim Rytin
m.r at prontomail.com