Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2012

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4

  • To: mathgroup at smc.vnet.net
  • Subject: [mg125131] Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4
  • From: Ray Koopman <koopman at sfu.ca>
  • Date: Thu, 23 Feb 2012 05:47:05 -0500 (EST)
  • Delivered-to: l-mathgroup@mail-archive0.wolfram.com
  • References: <ji2g4f$k8d$1@smc.vnet.net>

On Feb 22, 2:30 am, maria giovanna dainotti
<mariagiovannadaino... at yahoo.it> wrote:
> Dear Andy Ross,
> thanks for your suggestion. I applied the method
> DistributionFitTest[data1, data2,
> {"TestDataTable","SzekelyEnergy"},Method->{"MonteCarlo","MonteCarloSamples"->1000}]
> and it works, but when I try to minimize the function I have a problem.
>
> For[j = 1, j < n + 1,
>  fn = Function[{s, m},
>    x = Table[{Data1[[j, 1]] + s, Data1[[j, 2]] + m}, {j, 1, size}];
>    -2*Log[
>      DistributionFitTest[x, Data0, {"TestDataTable", "SzekelyEnergy"},
>         Method -> {"MonteCarlo", "MonteCarloSamples" -> 1000}][[1, 1,
>       2, 3]]]
>    ]]
>
> With Data1
> {{0.894606, 14.0865}, {0.811846, 13.7631}, {1.00287,
>   14.2577}, {0.798426, 13.8605}, {0.765556, 14.0297}, {1.06602,
>   14.144}, {0.965456, 14.2174}, {0.879326, 14.6311}, {0.831876,
>   14.006}, {0.695166, 14.2763}, {1.05657, 14.2558}, {1.05519,
>   13.7674}, {0.811626, 14.1137}, {0.910116, 13.9954}, {0.933376,
>   13.9232}, {0.870476, 14.2965}, {0.713326, 14.7321}, {1.13929,
>   13.9139}, {1.1878, 14.0376}, {0.998686, 13.9364}, {1.10588,
>   14.0096}, {1.06213, 14.2319}, {0.797846, 14.9199}, {0.768916,
>   14.5909}, {0.944476, 14.8957}, {1.09493, 14.6853}, {1.03171,
>   14.5485}, {0.838416, 15.1119}, {1.03875, 14.3372}, {0.695036,
>   15.0422}, {0.860336, 14.9023}, {0.903736, 15.1527}, {0.798196,
>   14.7923}, {0.695146, 15.5481}, {1.08506, 15.2012}, {1.07127,
>   15.2615}, {0.919576, 15.5636}, {1.01526, 15.4415}, {1.15206,
>   15.0784}, {0.844826, 15.6197}, {0.897866, 15.887}, {0.762816,
>   15.7318}, {0.867886, 15.743}, {0.991576, 15.7403}, {0.898346,
>   15.6946}, {0.996826, 15.6467}, {1.0194, 15.6681}, {0.830686,
>   15.5817}, {0.982806, 15.6863}, {0.883306, 16.3492}, {0.928786,
>   15.9975}, {0.932786, 15.9091}, {1.08748, 15.8808}, {0.963906,
>   16.4309}, {0.896236, 16.7413}, {0.750366, 16.7619}, {0.945936,
>   16.6388}, {1.064, 16.7059}, {0.947046, 17.0593}, {1.0258,
>   16.3197}, {0.973976, 16.9003}, {0.829206, 17.183}, {0.917956,
>   17.0182}, {0.954016, 16.8625}, {0.842906, 17.2803}, {1.10261,
>   17.0396}, {1.02155, 17.3819}, {0.671206, 17.5726}, {1.04596,
>   17.076}, {1.08985, 17.3047}, {1.05809, 17.2071}, {0.916836,
>   17.6237}, {0.815676, 17.5559}, {0.943866, 18.1943}, {1.02348,
>   16.8333}, {1.00346, 16.9493}, {1.15636, 17.0909}, {1.11935,
>   17.5003}}
>
> and Data0
>
> {{0.97304, 14.1829}, {0.98663, 14.1295}, {0.98284, 14.3172}, {0.81423,
>    14.3466}, {0.97303, 14.5966}, {0.87122, 14.8435}, {0.90252,
>   14.9036}, {0.81887, 15.1177}, {1.07722, 14.6849}, {0.86684,
>   15.6456}, {0.86664, 14.7034}, {0.78728, 15.0898}, {1.10336,
>   14.4085}, {1.12014, 14.7281}, {0.95923, 14.4988}, {0.89942,
>   15.3173}, {0.83841, 14.8422}, {0.99105, 14.8813}, {1.111,
>   14.5964}, {0.93255, 15.5019}, {1.03142, 14.8009}, {1.00661,
>   15.0827}, {0.93255, 15.3064}, {1.10023, 14.7189}, {0.8797,
>   15.1038}, {1.0013, 14.6755}, {0.87673, 15.0952}, {0.84131,
>   15.7345}, {1.06392, 15.3528}, {1.00138, 14.9835}, {0.77803,
>   15.4637}, {0.76795, 15.3611}, {0.98328, 15.1047}, {0.89193,
>   14.8321}, {0.72882, 15.909}, {0.77123, 15.7902}, {0.86218,
>   15.6637}, {0.84381, 15.536}, {0.99263, 15.8903}, {1.0805,
>   15.1453}, {0.85316, 15.7793}, {0.85186, 15.9119}, {1.10898,
>   15.7583}, {1.03365, 15.7393}, {0.84783, 16.1911}, {1.10979,
>   16.0031}, {1.05238, 16.015}, {0.90259, 16.4864}, {0.84963,
>   15.9818}, {1.09221, 16.0088}, {1.0443, 15.8326}, {0.8945,
>   16.1927}, {0.83015, 16.2776}, {0.7551, 16.5538}, {1.05947,
>   15.9138}, {1.06189, 15.6061}, {1.05889, 16.0743}, {0.85216,
>   16.1568}, {0.72597, 16.7657}, {0.93638, 16.3583}, {0.81968,
>   16.2711}, {0.95022, 16.3549}, {1.04536, 16.18}, {1.0786,
>   16.0967}, {0.9385, 15.8504}, {0.95024, 15.9338}, {0.76753,
>   17.1461}, {1.18224, 16.0437}, {0.96447, 16.4908}, {0.98235,
>   16.0892}, {1.06151, 16.699}, {0.79052, 16.5207}, {1.15863,
>   16.3223}, {1.00795, 16.2444}, {1.07284, 16.6536}, {1.04796,
>   16.865}, {0.84226, 16.7247}, {1.04712, 16.2673}}
>
> [...]

Here is an implementation of the Szekely-Energy comparison of your
two data sets. It saves some time by taking advantage of the fact
that your two sample sizes are equal. Perhaps more importantly, it
uses a centered orthonormal basis for the space of the pooled data.
This gives both variables equal opportunity to influence the test
statistic. If this is not done, the test statistic tends to ignore
variables with smaller variances.

Based on 10^6 samples, the approximate p-value is .0587 .

n1 = Length[data1 = {{0.894606, 14.0865}, ..., {1.11935, 17.5003}}]
n0 = Length[data0 = {{0.97304, 14.1829}, ..., {1.04712, 16.2673}}]
n = Length[data = Join[data1, data0]]

78
78
156

zz = Transpose@Orthogonalize[Transpose@data - Mean@data];
d = Outer[Norm[#1-#2]&, zz, zz, 1];
a = Total[Take[d, n1, -n0], 2]

883.529

reps = 1*^6
Timing[
  N@Total @ UnitStep[ Table[
  s = RandomSample[r = Range@n, n1]; r[[s]] = 0;
  t = SparseArray[r] /. SparseArray[_,_,_,x_] :> x[[3]];
  Total[d[[s,t]], 2], {reps}] - a] / reps ]

1000000

{3901.83, 0.058742}



  • Prev by Date: Re: Extensive replacement of trigonometric functions
  • Next by Date: Re: Executing a Notebook and saving the modified Notebook - all without the GUI
  • Previous by thread: Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4
  • Next by thread: Executing a Notebook and saving the modified Notebook - all without the GUI