Re: Shuffling 10^8 numbers: memory problems

*To*: mathgroup at smc.vnet.net*Subject*: [mg53300] Re: Shuffling 10^8 numbers: memory problems*From*: George Szpiro <george at netvision.net.il>*Date*: Wed, 5 Jan 2005 01:21:14 -0500 (EST)*References*: <crdm08$ra5$1@smc.vnet.net>*Sender*: owner-wri-mathgroup at wolfram.com

I realized that the problem probably starts when I write the results to the file. My computer has sufficient RAM to read the whole input file. I tried writing to the same file that I read from, but that did not solve the problem. Question: Is there a way to read the data that I am done with, i.e., LL[[j]] , to the output file and then delete it from memory? Since shuffling proceeds sequentially in the program below, there may be a way to do this every time 10,000 entries have been shuffled: write them to the output file, and free up memory. How? Thanks and best regards, George ----- Original Message ----- From: "George Szpiro" <george at netvision.net.il> To: mathgroup at smc.vnet.net Subject: [mg53300] Shuffling 10^8 numbers: memory problems > > The problem of quadratic time seems to have been solved by using > > aa = Developer`ToPackedArray[aa]; > > after reading in the data. > > But now the Mathematica kernel hits the limit again at 2*10^7 data: it > shuts down > MemoryInUse[] shows about 82 MB were used, but my machine has 704MB RAM > installed! What is happening? > > Is there a way to work around that one? I need to shuffle five times as > many integer numbers, i.e., I would need about 420 MB memory. > > Would it maybe help to start my machine in Safe Mode, so that no other > processes run, and thus get the use of nearly the full 704 MB memory? > (By the way, the 10^8 numbers are integers that are not larger than > about 1000 or 5000.) > > Thanks, > George > > The program I used: > > Clear[SSS,aa]; > SSS=20000000; (*I need to use 100000000*) > > aa=ReadList["c:GG.doc",Number,SSS]; > aa = Developer`ToPackedArray[aa]; > > SetAttributes[shuffleSet, HoldFirst]; > shuffleSet[LL_] := Module[{n=SSS, rand}, > Do[ > rand = Random[Integer, {j,n}]; > LL[[{j,rand}]] = LL[[{rand,j}]], > {j,n}]; > ] > > Timing[shuffleSet[aa]] > > <<Utilities`CleanSlate` > > OpenWrite["c:ShuffledGG.doc"]; > Put[OutputForm[aa],"c:ShuffledGG.doc"] > Close["c:ShuffledGG.doc"]; > > >