is there a better way to iterate this?
- To: mathgroup at smc.vnet.net
- Subject: [mg77820] is there a better way to iterate this?
- From: dantimatter <dantimatter at gmail.com>
- Date: Sun, 17 Jun 2007 06:13:05 -0400 (EDT)
Hello all,
I have a question about a specific problem I'm working now. I would
like to do 100000+ iterations of a series of operations, but the way
I'm doing it now is slow and memory intensive. I know there's a
better way to do it, but I just can't figure it out. The problem: I
have two lists of 21 sublists: listA and listB, and I would like to
shuffle the order of the sublists in listA around and then correlate
them with listB. Here's how I'm doing it now:
------------------------ begin inefficient code
-----------------------
randomizedlistA = Table[RandomPermutation[listA], {100000}];
listAlistBpairs =
Table[Table[
Transpose[{listB[[j]], randomizedlistA[[k, j]]},
{j, Length[listB]}], {k, Length[randomizedlistA]}];
foo =
Table[Table[
listAlistBpairs[[k, All, i]], {i,
Length[listAlistBpairs[[k, 1]]]}], {k,
Length[listAlistBpairs]}];
bar =
Table[N[Correlation[Flatten[foo[[i]], 1][[All,1]],
Flatten[foo[[i]], 1][[All, 2]], {i,
Length[foo]}];
------------------------ end inefficient code
--------------------------
The variable 'bar' should now be a list of 100000 correlation
coefficients. I realize this is probably bad Mathematica code, but I
wrote it quickly and it worked great for smaller numbers of
permutations. Can anyone suggest an easy way to do this better?
Also, though it's not written in that code, I'd like to take those
correlation coefficients and generate a probability distribution out
of them. For 10000 permutations it comes out as a beautiful
Gaussian; I can only assume it's going to look even better for
100000+ permutations. Any suggestions on how to generate a
probability distribution, and how to calculate P-values from that
distribution, would also be extremely helpful.
Many thanks,
Dan