Re: Standard deviations and Confidence intervals with respect to errors
- To: mathgroup at smc.vnet.net
- Subject: [mg101821] Re: Standard deviations and Confidence intervals with respect to errors
- From: Bill Rowe <readnews at sbcglobal.net>
- Date: Sat, 18 Jul 2009 04:50:51 -0400 (EDT)
On 7/16/09 at 8:20 AM, lehin.p at gmail.com (Alexey) wrote:
>Hello, I have a set of independent observations of two values (Y,X)
>with known (and different) standard errors in both values for each
>observation. These valueses are linearly dependent: Yi=a*Xi+b I need
>to calculate the unknown parameters {a,b} and confidence intervals
>with respect to known standard errors of observations. As I
>understand, the standard Mathematica's function LinearModelFit is
>designed only for the case when all observations have equal standard
>deviation and are normally distributed.
These are the standard assumptions for regression analysis. The
routines available in Mathematica version 7 do allow for some
deviation from these assumptions. Specifically, you can use the
Weights option in LinearModelFit to adjust for variations in the
standard deviation between observations. The usual procedure is
to specify the weight for each observation as 1/variance for
that observation.
For even more flexibility you can use GeneralizedLinearModelFit
which allows for probability distributions for the error term
different than a normal distribution. This function will also do
a weighted regression to allow for variations in the standard
deviation of observations.
However, none of the built-in routines are intended to address
the case where there are errors in X as well as errors in Y. If
you really do have significant errors in X you will need to
write your own fitting routine in order to get statistically
valid confidence limits.
>In my case it is known only that all observations have the same
>distribution that is supposed to be nearly-normal. But standard
>deviations for observations are significantly different and known.
Then it is quite possible the values computed using
LinearModelFit will be adequate. Surely, it is not reasonable to
expend considerable effort to meet some standard for
mathematical rigor if the end result is say a difference in the
third significant digit when your data is say only accurate to
two significant digits. If the error distribution is "nearly
normal" it could well be there is nothing to be gained by being
more rigorous. What level of rigor do you really need? What is
your end goal?
>And another question: how in this case the term "standard deviation"
>may be defined and calculated?
If you are simply asking how standard deviation is defined and
can be computed, then standard deviation is the square root of
variance. Look up Variance using the Documentation Center. In
the more information entry are details of how variance is
computed. Or in standard statistics language variance is the
expected value of (x-mu)^2 where x is the data sample and mu is
the mean of the data sample.
If this doesn't address your question about standard deviation,
you will need to re-phrase your question or expand on it.
If I have answered your question, that suggests to me you don't
have much knowledge of statistical computations/theory. If that
is true, trying to compute confidence limits where there are
significant errors in both x and y that are not from a normal
distribution is very much akin to trying to run a marathon when
you haven't learned to walk. In which case, you would be best
advised to study some good texts on statistics that should be
available in your local library. The documentation for various
statistical routines in Mathematica is no substitute for a good text.