Re: shuffling 10^8 numbers
- To: mathgroup at smc.vnet.net
- Subject: [mg53204] Re: [mg53180] shuffling 10^8 numbers
- From: János <janos.lobb at yale.edu>
- Date: Tue, 28 Dec 2004 23:12:38 -0500 (EST)
- References: <200412281130.GAA26970@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
First, try to set $HistoryLength to a needed finite number. That will free memory of Ins and Outs. Second, try to write back to file intermediate results and quit the kernel. That is a pain, but... On some OSs it does not work - like OSX - and even quitting Mathematica does not help to give back the memory. Rebooting helps all the time. Third, get a 64bit OS and 64bit Mathematica on it. János On Dec 28, 2004, at 6:30 AM, George Szpiro wrote: > Hi, > > I am trying to shuffle 10^8 numbers stored in the file GG.doc in the > root directory. (Size of GG.doc appros 360 MB) > > Accorrding to previous suggestions from this group I try to shuffle > them witht he following program: > > GG=OpenRead["c:\GG.doc"]; > AA=ReadList[GG]; > Timing[ > OrigList=Table[AA]; > p=RandomPermutation@Length@OrigList; > ShuffledList=OrigList[[p]]; > > > But the file is far too big. I can read it but then I get the > following error message: > > <<No more memory available. Mathematica kernel has shut down. Try > quitting other applications and then retry.>> > > No other programs are open, so I guess I am at the limit. Can anybody > suggest a workaround? Is there a possibility to shuffle numbers > without loading them all into memory simultaneously? > > NEW IDEA: I thought there might be a possibility of just reading one > single number each time from the file GG.doc, and putting them into a > randomly chosen slot in a new file. > > Any answeres greatly appreciated to: > george at netvision.net.il > > Thanks, > George > > ------------------------------------------ "The shortest route between two points is the middleman" Ayn Rand
- References:
- shuffling 10^8 numbers
- From: George Szpiro <george@netvision.net.il>
- shuffling 10^8 numbers