MathGroup Archive 2005

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Shuffling 10^8 numbers: memory problems

I realized that the problem probably starts when I write the results to the
file. My computer has sufficient RAM to read the whole input file. I tried
writing to the same file that I read from, but that did not solve the

Question: Is there a way to read the data that I am done with, i.e.,
LL[[j]] , to the output file and then delete it from memory? Since shuffling
proceeds sequentially in the program below, there may be a way to do this
every time 10,000 entries have been shuffled: write them to the output file,
and free up memory. How?

Thanks and best regards,

----- Original Message ----- 
From: "George Szpiro" <george at>
To: mathgroup at
Subject: [mg53300] Shuffling 10^8 numbers: memory problems

> The  problem of quadratic time seems to have been solved by using
> aa = Developer`ToPackedArray[aa];
> after reading in the data.
> But now the Mathematica kernel hits the limit again at 2*10^7 data: it
> shuts down
> MemoryInUse[] shows about 82 MB were used, but my machine has 704MB RAM
> installed! What is happening?
> Is there a way to work around that one? I need to shuffle five times as
> many integer numbers, i.e., I would need about 420 MB memory.
> Would it maybe help to start my machine in Safe Mode, so that no other
> processes run, and thus get the use of nearly the full 704 MB memory?
> (By the way, the 10^8 numbers are integers that are not larger than
> about 1000 or 5000.)
> Thanks,
> George
> The program I used:
> Clear[SSS,aa];
> SSS=20000000;   (*I need to use 100000000*)
> aa=ReadList["c:GG.doc",Number,SSS];
> aa = Developer`ToPackedArray[aa];
> SetAttributes[shuffleSet, HoldFirst];
> shuffleSet[LL_] := Module[{n=SSS, rand},
>    Do[
>        rand = Random[Integer, {j,n}];
>        LL[[{j,rand}]] = LL[[{rand,j}]],
>        {j,n}];
>    ]
> Timing[shuffleSet[aa]]
> <<Utilities`CleanSlate`
> OpenWrite["c:ShuffledGG.doc"];
> Put[OutputForm[aa],"c:ShuffledGG.doc"]
> Close["c:ShuffledGG.doc"];

  • Prev by Date: Re: Shuffling 10^8 numbers: memory problems
  • Next by Date: Re: shuffling 10^8 numbers
  • Previous by thread: Re: Shuffling 10^8 numbers: memory problems
  • Next by thread: shuffling 10^8 numbers