Re: Re: Memory Leak with KSubsets?
- To: mathgroup at smc.vnet.net
- Subject: [mg38525] Re: [mg38512] Re: Memory Leak with KSubsets?
- From: Kirk Reinholtz <kirk.reinholtz at jpl.nasa.gov>
- Date: Fri, 20 Dec 2002 23:40:52 -0500 (EST)
- References: <200212100917.EAA14948@smc.vnet.net> <at9e33$q6o$1@smc.vnet.net> <200212200928.EAA20542@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
I had a similar problem using combinatoria to do a bunch of transitive
closures: It grew ever larger in memory consumption, no matter what I
did, and I tried all the usual (check for accidential persistent
references to results, history list, etc) and I still didn't figure it
out. But, when I rewrote the combinatorica algorithm so that only one
form of result was maintained (adjacency list) instead of the several
forms it maintains in a -Graph- thing, the leak went away. I have no
proof, but it felt like results were being held on too long either in
combinatorica or mm itself. I spent a couple of hours in the
conbinatiorica code looking for the problem (that's how I ended up doing
the rewrite) but still I didn't figure it out. But it was interesting
that using the same algorithms, module structure, etc, I got it to run
w/o leaking by eliminating some of the items being passed around.
Bern wrote:
>
> Hello group,
>
> Skiena pointed out that I was too quick to blame RandomKSubset and
> KSubsets for this problem, in fact it happens for any such finction like
> Take. He pointed out that the problem I illustrated with
> memorySwallower[] could be fixed by putting in some other command after
> the creation of the Table. However I have since realized that while his
> fix is correct for memorySwallower, it did not yet fix the problem I am
> having with the original program. In fact I am using KSubsets and
> RandomKSubset in a large Module and had assumed that they were causing
> the problem I described in memorySwallower. Putting Skiena's dummy
> variables in my original program helped, but did not solve the problem.
> I would post the actual program/Module, except that it is very large
> and of course much of it is irrelevant. Some have suggested that
> setting $HistoryLength to a more reasonable number would fix the
> problem, but I had done this long ago - and to no avail. I have checked
> all of the variables to make sure that they are all declared local in
> the beginning of Module, this doesn't seem to be the problem. I have
> the problem in 4.1 and 4.2
>
> It seems to me that part of the issue is that Module doesn't release
> all of its memory when finished. For instance, why, in my original
> memorySwallower, was there even the issue of memory being swallowed,
> prior to the inclusion of 'dummy' that fixed it? Shouldn't the exiting
> of a Module release any memory used by local variables? And why do we
> have the problem even if we use 'Do' instead of 'Table'???
>
> Ok, thanks again,
>
> Bern
>
> > MemoryInUse[]
> > Get["DiscreteMath`Combinatorica`"]
> > MemoryInUse[]
> > Out[1]=
> > 1398624
> > Out[3]=
> > 1956536
> >
> > memorySwallower[S_,n_]:=
> > Module[{data,subsamples},Print[ToString[MemoryInUse[]]];
> > Table[data=Table[{i,j},{i,1000},{j,1+m}];
> > subsamples=Table[RandomKSubset[data,S],{100}],{m,n}];
> > Print[ToString[MemoryInUse[]]]]
> >
> >
> > memorySwallower[3,100]
> >
> > 15497936
> > 29833672
>
> Skiena wrote:
> Well, I can confirm that the problem isn't Combinatorica, since
> the following:
>
> memorySwallower[S_, n_] :=
> Module[{data, subsamples}, Print[ToString[MemoryInUse[]]];
> Table[data = Table[{i, j}, {i, 1000}, {j, 1 + m}];
> subsamples = Table[Take[data, S], {100}], {m, n}];
> Print[ToString[MemoryInUse[]]]]
>
> has the same memory consumption behavior. (Replaced RandomKSubset
> with Take, which returns a list of the same length...) You can
> at least pass that much along to the user to exonerate yourself... :-)
>
> Actually, looking at this example some more, the problem is simply
> that the result (the full table, which is 13 MB or so in size)
> is getting cached in the command output history. At the time of the
> second MemoryInUse call, the table has not been deleted; and because
> Print returns Null, the value of the table is left behind to get
> cached in Out (the value that gets stored in Out is the last non-Null
> value from the compound expression). So, for example, if you were
> to evaluate
>
> memorySwallower[3, 100]; dummy;
>
> instead of just the memorySwallower command, 'dummy' gets put into
> the output history instead of the table, and successive calls to this
> show the memory does not continue increasing. (The second call is
> still 15 MB higher than the first call, since the table hasn't been
> deleted at that point...) Equivalently, if you write e.g.:
- References:
- Memory Leak with KSubsets?
- From: "Arny" <someone@somewhere.sometime>
- Re: Memory Leak with KSubsets?
- From: Bern <bgress@mail.DOTucr.DOT.edu>
- Memory Leak with KSubsets?