Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Shuffling 10^8 numbers: memory problems

• To: mathgroup at smc.vnet.net
• Subject: [mg53295] Shuffling 10^8 numbers: memory problems
• From: George Szpiro <george at netvision.net.il>
• Date: Tue, 4 Jan 2005 03:14:23 -0500 (EST)
• Sender: owner-wri-mathgroup at wolfram.com

The  problem of quadratic time seems to have been solved by using

aa = Developer`ToPackedArray[aa];

after reading in the data.

But now the Mathematica kernel hits the limit again at 2*10^7 data: it shuts down
MemoryInUse[] shows about 82 MB were used, but my machine has 704MB RAM 
installed! What is happening?

Is there a way to work around that one? I need to shuffle five times as
many integer numbers, i.e., I would need about 420 MB memory.

Would it maybe help to start my machine in Safe Mode, so that no other
processes run, and thus get the use of nearly the full 704 MB memory?
(By the way, the 10^8 numbers are integers that are not larger than
about 1000 or 5000.)

Thanks,
 George

The program I used:

Clear[SSS,aa];
SSS=20000000;   (*I need to use 100000000*)

aa=ReadList["c:GG.doc",Number,SSS];
aa = Developer`ToPackedArray[aa];

SetAttributes[shuffleSet, HoldFirst];
shuffleSet[LL_] := Module[{n=SSS, rand},
    Do[
        rand = Random[Integer, {j,n}];
        LL[[{j,rand}]] = LL[[{rand,j}]],
        {j,n}];
    ]

Timing[shuffleSet[aa]]

<<Utilities`CleanSlate`

 OpenWrite["c:ShuffledGG.doc"];
Put[OutputForm[aa],"c:ShuffledGG.doc"]
Close["c:ShuffledGG.doc"];

• Follow-Ups:
  • Re: Shuffling 10^8 numbers: memory problems
    • From: János <janos.lobb@yale.edu>