Services & Resources / Wolfram Forums
-----
 /
MathGroup Archive
2004
*January
*February
*March
*April
*May
*June
*July
*August
*September
*October
*November
*December
*Archive Index
*Ask about this page
*Print this page
*Give us feedback
*Sign up for the Wolfram Insider

MathGroup Archive 2004

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: shuffling 10^8 numbers

  • To: mathgroup at smc.vnet.net
  • Subject: [mg53204] Re: [mg53180] shuffling 10^8 numbers
  • From: János <janos.lobb at yale.edu>
  • Date: Tue, 28 Dec 2004 23:12:38 -0500 (EST)
  • References: <200412281130.GAA26970@smc.vnet.net>
  • Sender: owner-wri-mathgroup at wolfram.com

First, try to set $HistoryLength to a needed finite number. That will 
free memory of Ins and Outs.
Second, try to write back to file intermediate results and quit the 
kernel. That is a pain, but... On some OSs it does not work - like OSX 
- and even quitting Mathematica does not help to give back the memory. 
Rebooting helps all the time.
Third, get a 64bit OS and 64bit Mathematica on it.

János
On Dec 28, 2004, at 6:30 AM, George Szpiro wrote:



> Hi,
>
> I am trying to shuffle 10^8 numbers stored in the file GG.doc in the 
> root directory. (Size of GG.doc appros 360 MB)
>
> Accorrding to previous suggestions from this group I try to shuffle 
> them witht he following program:
>
> GG=OpenRead["c:\GG.doc"];
> AA=ReadList[GG];
>  Timing[
>  OrigList=Table[AA];
>  p=RandomPermutation@Length@OrigList;
>  ShuffledList=OrigList[[p]];
>
>
> But the file is far too big. I can read it but then I get the 
> following error message:
>
> <<No more memory available. Mathematica kernel has shut down. Try 
> quitting other applications and then retry.>>
>
> No other programs are open, so I guess I am at the limit. Can anybody 
> suggest a workaround? Is there a possibility to shuffle numbers 
> without loading them all into memory simultaneously?
>
> NEW IDEA: I thought there might be a possibility of just reading one 
> single number each time from the file GG.doc, and putting them into a 
> randomly chosen slot in a new file.
>
> Any answeres greatly appreciated to:
> george at netvision.net.il
>
> Thanks,
> George
>
>
------------------------------------------
"The shortest route between two points is the middleman" Ayn Rand


  • Prev by Date: Re: N[] does not work inside Replace[] ?
  • Next by Date: Re: Re: Condition , FullSimplify
  • Previous by thread: Re: shuffling 10^8 numbers
  • Next by thread: Re: shuffling 10^8 numbers