Re: question: fitting a distribution from quantiles

• To: mathgroup at smc.vnet.net
• Subject: [mg126466] Re: question: fitting a distribution from quantiles
• From: LÃszlÃ SÃndor <sandorl at gmail.com>
• Date: Sat, 12 May 2012 04:55:54 -0400 (EDT)
• Delivered-to: l-mathgroup@mail-archive0.wolfram.com

```Oh, shame, sorry about this. And thanks, of course!

Laszlo

On Fri, May 11, 2012 at 12:01 PM, Darren Glosemeyer <darreng at wolfram.com>wrote:

>  There's a typo in the code using CDF directly. For the cdf, you need to
> use CDF[ParetoDistribution[k, a], x], then it will work fine.
>
> Darren Glosemeyer
> Wolfram Research
>
>
> On 5/11/2012 10:55 AM, L=C3=A1szl=C3=B3 S=C3=A1ndor wrote:
>
> Thank you, Darren!
>
>  I realized soon (much before the delay cause my the moderation of the
> list) that I could fit a CDF. This even works with a ridiculously
> aggregated data, e.g. only two (inverse) quantiles for a Pareto
> distribution. However, FindFit did not work with Mathematica's
> representation of the CDF (conditions?), only a hard-coded one.
>
>  But before I paste my output (with a lengthy error message) below, let
> me ask another question: What exactly are the benefits of keeping a
> distribution object in the background? Am I just as well off with a
> (smoothed) CDF and plugging it or transformations and integrals everywhere?
>
>  Basically, I want to use an empirical distribution in three ways:
> -- "keep it as it is" (though it must be smoothed / approximated) as I do
> need PDFs even though, as any real data, it comes discrete
> -- fit a parametric distribution and use that everywhere where I would
> have used the empirical
> -- fit a mixture of parametric distributions (actually, it might be a
> special mixture: I might concatenate two different (truncated) CDF for
> different parts -- real incomes have a Pareto right tail but an obviously
> non-Pareto bottom.
>
>  Is this  good idea to try to keep these as distributions, or as most of
> my calculation will need to numeric anyway, I can give up early and use the
> CDFs?
>
>  Thanks!
>
>  Now the output for yesterday:
>
>  originalecdf =
> {{500000,0.0182},{1000000,0.1003},{1500000,0.2487},{2000000,0.3871},{4000000,0.6802}}
> ecdf = {{2000000,0.3871},{4000000,0.6802}}
> FindFit[ecdf,CDF[ParetoDistribution[k,a]],{k,a},x]
>
>  FindFit::nrlnum: The function value
> {-0.3871+Function[\[FormalX],\[Piecewise] 1. +Times[<<2>>] \[FormalX]>==
k
> 0. True
>
>
>
>  ,Listable],-0.6802+Function[\[FormalX],\[Piecewise] 1. +Times[<<2>>]
> \[FormalX]>=k
> 0. True
>
>
>
>  ,Listable]}
>  is not a list of real numbers with dimensions {2} at {k,a} = {1.,1.}. >>
>
>  FindFit[ecdf,1-(x / k)^(-a),{k,a},x]
> {k->1.18709*10^6,a->0.938482}
>
>
>
> On Fri, May 11, 2012 at 11:37 AM, Darren Glosemeyer <darreng at wolfram.com> wrote:
>
>> On 5/10/2012 11:14 PM, L=C3=A1szl=C3=B3 S=C3=A1ndor wrote:
>>
>>> Hi all,
>>>
>>> I have a project (with Mathematica 8) where the first step would be to
>>> get the distribution describing my "data" which actually only have
>>> quantiles (or worse: frequencies for arbitrary bins).
>>> EstimatedDistribution[] looks promising, but I don't know how to feed in
>>> this kind of data. Please let me know if you know a fast way.
>>>
>>> Thank!
>>>
>>>
>>>
>> There isn't enough information in your data for the types of estimation
>> done by EstimatedDistribution.
>>
>> The type of information you have in your data would lend itself well to a
>> least squares fit to the cdf of the distribution. As an example, let's take
>> this data:
>>
>>
>> In[1]:= data = BlockRandom[SeedRandom[1234];
>>
>> We can use Min and Max to see the range of values and then bin within
>> that range to construct cutoff and frequency data.
>>
>> In[2]:= {Min[data], Max[data]}
>>
>> Out[2]= {13.7834, 112.429}
>>
>>
>> Here, xvals are the cutoffs and counts are the bin frequencies.
>>
>> In[3]:= {xvals, counts} = HistogramList[data, {{0, 15, 20, 50, 100, 120}}]
>>
>> Out[3]= {{0, 15, 20, 50, 100, 120}, {1, 6, 55, 37, 1}}
>>
>>
>> We can get the accumulated probabilities as follows.
>>
>> In[4]:= probs = Accumulate[counts]/Length[data]
>>
>>          1    7   31  99
>> Out[4]= {---, ---, --, ---, 1}
>>         100  100  50  100
>>
>>
>> The analogue of your quantile values would be the right endpoints,
>> Rest[xvals].
>>
>> In[5]:= quantiles = Rest[xvals]
>>
>> Out[5]= {15, 20, 50, 100, 120}
>>
>>
>> Now we can use the quantiles as the x values and the cdf values as the y
>> values for a least squares fitting to the CDF (parameters may need starting
>> values in general, but defaults worked fine in this case):
>>
>> b], x], {a, b}, x]
>>
>> Out[6]= {a -> 5.24009, b -> 8.88512}
>>
>>
>> Given that we know that the data don't extend to the right limit of a
>> gamma's support (gammas can be any positive values), we may want to adjust
>> the cdf values a bit. The following will shift all the cdf values by
>> 1/(2*numberOfDataPoints) in this particular case:
>>
>> In[7]:= FindFit[Transpose[{quantiles, probs - 1/(2 Length[data])}],
>>         CDF[GammaDistribution[a, b], x], {a, b}, x]
>>
>> Out[7]= {a -> 5.3696, b -> 8.73319}
>>
>>
>> Darren Glosemeyer
>> Wolfram Research
>>
>
>
>

```

• Prev by Date: Re: question: fitting a distribution from quantiles
• Next by Date: Saving PDE Derivatives
• Previous by thread: Re: question: fitting a distribution from quantiles
• Next by thread: Condition Generation in Integrate