Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Re: Memory Leak with KSubsets?

• To: mathgroup at smc.vnet.net
• Subject: [mg38512] Re: Memory Leak with KSubsets?
• From: Bern <bgress at mail.DOTucr.DOT.edu>
• Date: Fri, 20 Dec 2002 04:28:10 -0500 (EST)
• Organization: University of California, Riverside
• References: <200212100917.EAA14948@smc.vnet.net> <at9e33$q6o$1@smc.vnet.net>
• Sender: owner-wri-mathgroup at wolfram.com

Hello group,

Skiena pointed out that I was too quick to blame RandomKSubset and 
KSubsets for this problem, in fact it happens for any such finction like 
Take.  He pointed out that the problem I illustrated with 
memorySwallower[] could be fixed by putting in some other command after 
the creation of the Table.  However I have since realized that while his 
fix is correct for memorySwallower, it did not yet fix the problem I am 
having with the original program.  In fact I am using KSubsets and 
RandomKSubset in a large Module and had assumed that they were causing 
the problem I described in memorySwallower.  Putting Skiena's dummy 
variables in my original program helped, but did not solve the problem. 
  I would post the actual program/Module, except that it is very large 
and  of course much of it is irrelevant.  Some have suggested that 
setting $HistoryLength to a more reasonable number would fix the 
problem, but I had done this long ago - and to no avail. I have checked 
all of the variables to make sure that they are all declared local in 
the beginning of Module, this doesn't seem to be the problem.  I have 
the problem in 4.1 and 4.2

   It seems to me that part of the issue is that Module doesn't release 
all of its memory when finished.  For instance, why, in my original 
memorySwallower, was there even the issue of memory being swallowed, 
prior to the inclusion of 'dummy' that fixed it?  Shouldn't the exiting 
of a Module release any memory used by local variables?  And why do we 
have the problem even if we use 'Do' instead of 'Table'???

Ok, thanks again,

Bern

 > MemoryInUse[]
 > Get["DiscreteMath`Combinatorica`"]
 > MemoryInUse[]
 > Out[1]=
 > 1398624
 > Out[3]=
 > 1956536
 >
 > memorySwallower[S_,n_]:=
 >   Module[{data,subsamples},Print[ToString[MemoryInUse[]]];
 >     Table[data=Table[{i,j},{i,1000},{j,1+m}];
 >       subsamples=Table[RandomKSubset[data,S],{100}],{m,n}];
 >     Print[ToString[MemoryInUse[]]]]
 >
 >
 > memorySwallower[3,100]
 >
 > 15497936
 > 29833672

Skiena wrote:
Well, I can confirm that the problem isn't Combinatorica, since
the following:

memorySwallower[S_, n_] :=
   Module[{data, subsamples}, Print[ToString[MemoryInUse[]]];
     Table[data = Table[{i, j}, {i, 1000}, {j, 1 + m}];
       subsamples = Table[Take[data, S], {100}], {m, n}];
     Print[ToString[MemoryInUse[]]]]

has the same memory consumption behavior. (Replaced RandomKSubset
with Take, which returns a list of the same length...) You can
at least pass that much along to the user to exonerate yourself... :-)

Actually, looking at this example some more, the problem is simply
that the result (the full table, which is 13 MB or so in size)
is getting cached in the command output history. At the time of the
second MemoryInUse call, the table has not been deleted; and because
Print returns Null, the value of the table is left behind to get
cached in Out (the value that gets stored in Out is the last non-Null
value from the compound expression). So, for example, if you were
to evaluate

memorySwallower[3, 100]; dummy;

instead of just the memorySwallower command, 'dummy' gets put into
the output history instead of the table, and successive calls to this
show the memory does not continue increasing. (The second call is
still 15 MB higher than the first call, since the table hasn't been
deleted at that point...) Equivalently, if you write e.g.:

• Follow-Ups:
  • Re: Re: Memory Leak with KSubsets?
    • From: Kirk Reinholtz <kirk.reinholtz@jpl.nasa.gov>

• References:
  • Memory Leak with KSubsets?
    • From: "Arny" <someone@somewhere.sometime>