Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4
- To: mathgroup at smc.vnet.net
- Subject: [mg125131] Re: Kolmogorov Smirnov in two or more dimensions is in Mathematica 8.0.4
- From: Ray Koopman <koopman at sfu.ca>
- Date: Thu, 23 Feb 2012 05:47:05 -0500 (EST)
- Delivered-to: l-mathgroup@mail-archive0.wolfram.com
- References: <ji2g4f$k8d$1@smc.vnet.net>
On Feb 22, 2:30 am, maria giovanna dainotti
<mariagiovannadaino... at yahoo.it> wrote:
> Dear Andy Ross,
> thanks for your suggestion. I applied the method
> DistributionFitTest[data1, data2,
> {"TestDataTable","SzekelyEnergy"},Method->{"MonteCarlo","MonteCarloSamples"->1000}]
> and it works, but when I try to minimize the function I have a problem.
>
> For[j = 1, j < n + 1,
> fn = Function[{s, m},
> x = Table[{Data1[[j, 1]] + s, Data1[[j, 2]] + m}, {j, 1, size}];
> -2*Log[
> DistributionFitTest[x, Data0, {"TestDataTable", "SzekelyEnergy"},
> Method -> {"MonteCarlo", "MonteCarloSamples" -> 1000}][[1, 1,
> 2, 3]]]
> ]]
>
> With Data1
> {{0.894606, 14.0865}, {0.811846, 13.7631}, {1.00287,
> 14.2577}, {0.798426, 13.8605}, {0.765556, 14.0297}, {1.06602,
> 14.144}, {0.965456, 14.2174}, {0.879326, 14.6311}, {0.831876,
> 14.006}, {0.695166, 14.2763}, {1.05657, 14.2558}, {1.05519,
> 13.7674}, {0.811626, 14.1137}, {0.910116, 13.9954}, {0.933376,
> 13.9232}, {0.870476, 14.2965}, {0.713326, 14.7321}, {1.13929,
> 13.9139}, {1.1878, 14.0376}, {0.998686, 13.9364}, {1.10588,
> 14.0096}, {1.06213, 14.2319}, {0.797846, 14.9199}, {0.768916,
> 14.5909}, {0.944476, 14.8957}, {1.09493, 14.6853}, {1.03171,
> 14.5485}, {0.838416, 15.1119}, {1.03875, 14.3372}, {0.695036,
> 15.0422}, {0.860336, 14.9023}, {0.903736, 15.1527}, {0.798196,
> 14.7923}, {0.695146, 15.5481}, {1.08506, 15.2012}, {1.07127,
> 15.2615}, {0.919576, 15.5636}, {1.01526, 15.4415}, {1.15206,
> 15.0784}, {0.844826, 15.6197}, {0.897866, 15.887}, {0.762816,
> 15.7318}, {0.867886, 15.743}, {0.991576, 15.7403}, {0.898346,
> 15.6946}, {0.996826, 15.6467}, {1.0194, 15.6681}, {0.830686,
> 15.5817}, {0.982806, 15.6863}, {0.883306, 16.3492}, {0.928786,
> 15.9975}, {0.932786, 15.9091}, {1.08748, 15.8808}, {0.963906,
> 16.4309}, {0.896236, 16.7413}, {0.750366, 16.7619}, {0.945936,
> 16.6388}, {1.064, 16.7059}, {0.947046, 17.0593}, {1.0258,
> 16.3197}, {0.973976, 16.9003}, {0.829206, 17.183}, {0.917956,
> 17.0182}, {0.954016, 16.8625}, {0.842906, 17.2803}, {1.10261,
> 17.0396}, {1.02155, 17.3819}, {0.671206, 17.5726}, {1.04596,
> 17.076}, {1.08985, 17.3047}, {1.05809, 17.2071}, {0.916836,
> 17.6237}, {0.815676, 17.5559}, {0.943866, 18.1943}, {1.02348,
> 16.8333}, {1.00346, 16.9493}, {1.15636, 17.0909}, {1.11935,
> 17.5003}}
>
> and Data0
>
> {{0.97304, 14.1829}, {0.98663, 14.1295}, {0.98284, 14.3172}, {0.81423,
> 14.3466}, {0.97303, 14.5966}, {0.87122, 14.8435}, {0.90252,
> 14.9036}, {0.81887, 15.1177}, {1.07722, 14.6849}, {0.86684,
> 15.6456}, {0.86664, 14.7034}, {0.78728, 15.0898}, {1.10336,
> 14.4085}, {1.12014, 14.7281}, {0.95923, 14.4988}, {0.89942,
> 15.3173}, {0.83841, 14.8422}, {0.99105, 14.8813}, {1.111,
> 14.5964}, {0.93255, 15.5019}, {1.03142, 14.8009}, {1.00661,
> 15.0827}, {0.93255, 15.3064}, {1.10023, 14.7189}, {0.8797,
> 15.1038}, {1.0013, 14.6755}, {0.87673, 15.0952}, {0.84131,
> 15.7345}, {1.06392, 15.3528}, {1.00138, 14.9835}, {0.77803,
> 15.4637}, {0.76795, 15.3611}, {0.98328, 15.1047}, {0.89193,
> 14.8321}, {0.72882, 15.909}, {0.77123, 15.7902}, {0.86218,
> 15.6637}, {0.84381, 15.536}, {0.99263, 15.8903}, {1.0805,
> 15.1453}, {0.85316, 15.7793}, {0.85186, 15.9119}, {1.10898,
> 15.7583}, {1.03365, 15.7393}, {0.84783, 16.1911}, {1.10979,
> 16.0031}, {1.05238, 16.015}, {0.90259, 16.4864}, {0.84963,
> 15.9818}, {1.09221, 16.0088}, {1.0443, 15.8326}, {0.8945,
> 16.1927}, {0.83015, 16.2776}, {0.7551, 16.5538}, {1.05947,
> 15.9138}, {1.06189, 15.6061}, {1.05889, 16.0743}, {0.85216,
> 16.1568}, {0.72597, 16.7657}, {0.93638, 16.3583}, {0.81968,
> 16.2711}, {0.95022, 16.3549}, {1.04536, 16.18}, {1.0786,
> 16.0967}, {0.9385, 15.8504}, {0.95024, 15.9338}, {0.76753,
> 17.1461}, {1.18224, 16.0437}, {0.96447, 16.4908}, {0.98235,
> 16.0892}, {1.06151, 16.699}, {0.79052, 16.5207}, {1.15863,
> 16.3223}, {1.00795, 16.2444}, {1.07284, 16.6536}, {1.04796,
> 16.865}, {0.84226, 16.7247}, {1.04712, 16.2673}}
>
> [...]
Here is an implementation of the Szekely-Energy comparison of your
two data sets. It saves some time by taking advantage of the fact
that your two sample sizes are equal. Perhaps more importantly, it
uses a centered orthonormal basis for the space of the pooled data.
This gives both variables equal opportunity to influence the test
statistic. If this is not done, the test statistic tends to ignore
variables with smaller variances.
Based on 10^6 samples, the approximate p-value is .0587 .
n1 = Length[data1 = {{0.894606, 14.0865}, ..., {1.11935, 17.5003}}]
n0 = Length[data0 = {{0.97304, 14.1829}, ..., {1.04712, 16.2673}}]
n = Length[data = Join[data1, data0]]
78
78
156
zz = Transpose@Orthogonalize[Transpose@data - Mean@data];
d = Outer[Norm[#1-#2]&, zz, zz, 1];
a = Total[Take[d, n1, -n0], 2]
883.529
reps = 1*^6
Timing[
N@Total @ UnitStep[ Table[
s = RandomSample[r = Range@n, n1]; r[[s]] = 0;
t = SparseArray[r] /. SparseArray[_,_,_,x_] :> x[[3]];
Total[d[[s,t]], 2], {reps}] - a] / reps ]
1000000
{3901.83, 0.058742}