Shuffling 10^8 numbers: memory problems

*To*: mathgroup at smc.vnet.net*Subject*: [mg53295] Shuffling 10^8 numbers: memory problems*From*: George Szpiro <george at netvision.net.il>*Date*: Tue, 4 Jan 2005 03:14:23 -0500 (EST)*Sender*: owner-wri-mathgroup at wolfram.com

The problem of quadratic time seems to have been solved by using aa = Developer`ToPackedArray[aa]; after reading in the data. But now the Mathematica kernel hits the limit again at 2*10^7 data: it shuts down MemoryInUse[] shows about 82 MB were used, but my machine has 704MB RAM installed! What is happening? Is there a way to work around that one? I need to shuffle five times as many integer numbers, i.e., I would need about 420 MB memory. Would it maybe help to start my machine in Safe Mode, so that no other processes run, and thus get the use of nearly the full 704 MB memory? (By the way, the 10^8 numbers are integers that are not larger than about 1000 or 5000.) Thanks, George The program I used: Clear[SSS,aa]; SSS=20000000; (*I need to use 100000000*) aa=ReadList["c:GG.doc",Number,SSS]; aa = Developer`ToPackedArray[aa]; SetAttributes[shuffleSet, HoldFirst]; shuffleSet[LL_] := Module[{n=SSS, rand}, Do[ rand = Random[Integer, {j,n}]; LL[[{j,rand}]] = LL[[{rand,j}]], {j,n}]; ] Timing[shuffleSet[aa]] <<Utilities`CleanSlate` OpenWrite["c:ShuffledGG.doc"]; Put[OutputForm[aa],"c:ShuffledGG.doc"] Close["c:ShuffledGG.doc"];

**Follow-Ups**:**Re: Shuffling 10^8 numbers: memory problems***From:*János <janos.lobb@yale.edu>