[Date Index] [Thread Index] [Author Index]

Re: shuffling 10^8 numbers

• To: mathgroup at smc.vnet.net
• Subject: [mg53204] Re: [mg53180] shuffling 10^8 numbers
• From: János <janos.lobb at yale.edu>
• Date: Tue, 28 Dec 2004 23:12:38 -0500 (EST)
• References: <200412281130.GAA26970@smc.vnet.net>
• Sender: owner-wri-mathgroup at wolfram.com

First, try to set $HistoryLength to a needed finite number. That will 
free memory of Ins and Outs.
Second, try to write back to file intermediate results and quit the 
kernel. That is a pain, but... On some OSs it does not work - like OSX 
- and even quitting Mathematica does not help to give back the memory. 
Rebooting helps all the time.
Third, get a 64bit OS and 64bit Mathematica on it.

János
On Dec 28, 2004, at 6:30 AM, George Szpiro wrote:



> Hi,
>
> I am trying to shuffle 10^8 numbers stored in the file GG.doc in the 
> root directory. (Size of GG.doc appros 360 MB)
>
> Accorrding to previous suggestions from this group I try to shuffle 
> them witht he following program:
>
> GG=OpenRead["c:\GG.doc"];
> AA=ReadList[GG];
>  Timing[
>  OrigList=Table[AA];
>  p=RandomPermutation@Length@OrigList;
>  ShuffledList=OrigList[[p]];
>
>
> But the file is far too big. I can read it but then I get the 
> following error message:
>
> <<No more memory available. Mathematica kernel has shut down. Try 
> quitting other applications and then retry.>>
>
> No other programs are open, so I guess I am at the limit. Can anybody 
> suggest a workaround? Is there a possibility to shuffle numbers 
> without loading them all into memory simultaneously?
>
> NEW IDEA: I thought there might be a possibility of just reading one 
> single number each time from the file GG.doc, and putting them into a 
> randomly chosen slot in a new file.
>
> Any answeres greatly appreciated to:
> george at netvision.net.il
>
> Thanks,
> George
>
>
------------------------------------------
"The shortest route between two points is the middleman" Ayn Rand

• References:
  • shuffling 10^8 numbers
    • From: George Szpiro <george@netvision.net.il>