Re: [long] Mathematica 5.1 and memory: any garbage collection?
- To: mathgroup at smc.vnet.net
- Subject: [mg59738] Re: [mg59697] [long] Mathematica 5.1 and memory: any garbage collection?
- From: Chris Chiasson <chris.chiasson at gmail.com>
- Date: Fri, 19 Aug 2005 04:32:07 -0400 (EDT)
- References: <200508180416.AAA08516@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
Someone posted about garbage collection a while ago. I think what
ended up happening was that they forgot to clear the Out symbol. No
doubt someone else will also post the method to clear everything from
the Kernel (via a package).
On 8/18/05, some.poster at use.net <some.poster at use.net> wrote:
> Hi,
>
> I've been using Mathematica v5.1 under XP for some time now and found it largely
> improved over previous versions, at least for my problems.
>
> Only recently did I begin to handle some real data: rather large List
> objects. I have to routinely select objects within lists as large as, say,
> 46000 * 512 * 20 items, items themselves not being atomic.
>
> Of course I never expected to be able to use functionnal style for dealing
> with such large guys on a 1Gb RAM PC: an outer For loop hands one object
> (out of 46000) at a time, then functionnal style handling follows. In this
> process I create variable-sized intermediate objects that are either
> accumulated in decimated lists or discarded altogether.
>
> It turns out that my typical intermediate objects eat up to 2.5 Mb each.
> This sounds a little of a memory hog, but anyway, I can live with that.
>
> My problem lies with the fact that memory used at intermediate stage seems
> never be freed, even whith explicit Clear, Remove or any assignment to {} or
> . in whatever order.
>
> How can one recover the memory freed (if some is ever effectively freed)? Is
> there _any_ garbage collection in the guts of this gazillion dollars piece
> of code?
>
> I have no problem with the idea of regularly invoking some function to
> compact the heap (should the kernel be unable to perform it by itself). The
> question is: what's its name?
>
> How do you reclaim memory, you serious users who do computations on large
> objects with numerous intermediate values?
>
>
> At the end, I tried this, from a fresh kernel load:
>
> In[1]:=
> $ProductInformation
>
> Out[1]=
> {ProductIDName\[Rule]Mathematica,
> ProductKernelName\[Rule]Mathematica 5.1 Kernel,
> ProductVersion\[Rule]5.1 for Microsoft Windows (October 25, 2004),
> ProductVersionNumber\[Rule]5.1}
>
> In[2]:=
> MemoryInUse[]
>
> Out[2]=
> 2124904
>
> In[3]:=
> (* let's create a moderate-size object *)
> try=Tuples[{a,b,c,d,e,f},7];
> ByteCount[try]
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[4]=
> 14556704
>
> Out[5]=
> {16684640,16685792}
>
> In[6]:=
> (* simple mapped function *)
> res=Map[2*#&,try];
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[7]=
> {93949328,103186832} (* Ouch ! So much for a _simple_ test *)
>
> In[8]:=
> (* forgetting a large expression doesn't free _any_ memory *)
> try={};
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[9]=
> {93951056,103186832}
>
> In[10]:=
> (* no more success this way *)
> try=.;
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[11]=
> {93952648,103186832}
>
> In[12]:=
> (* ... neither a sweep does *)
> Clear[try];
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[13]=
> {93954320,103186832}
>
> In[14]:=
> (* ... nor a complete wash *)
> Remove[try];
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[15]=
> {93955944,103186832}
>
> In[16]:=
> (* freeing the result also seems beyond possible *)
> res={};
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[17]=
> {93958112,103186832}
>
> In[18]:=
> (* still no luck, dad *)
> res=.;
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[19]=
> {93958952,103186832}
>
> In[20]:=
> (* ahem, what _is_ Clear supposed to do? *)
> Clear[res];
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[21]=
> {93960448,103186832}
>
> In[22]:=
> (* ahem, what _is_ Remove supposed to do? *)
> Remove[res];
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[23]=
> {93961936,103186832}
>
> In[24]:=
> (* sharing common subexpressions somehow works, slow as hell *)
>
> (* please note: even after removing all values and symbols in all
> possible ways, try and res are _still_ somewhere in memory since
> Share _really_ does something! *)
>
> (* Share should share nothing as we don't have any object stored, safe
> perhaps symbols a..f *)
>
> (* when we use larger objects or more intermediate results like res
> we get the kernel to shutdown very quickly at this rate *)
> Share[]
> {MemoryInUse[],MaxMemoryUsed[]}
>
> Out[24]=
> 62824872
>
> Out[25]=
> {31138184,103186832}
>
> (* did you expect to see about the same value as on the first call above? *)
>
>
--
Chris Chiasson
http://chrischiasson.com/
1 (810) 265-3161
- References:
- [long] Mathematica 5.1 and memory: any garbage collection?
- From: some.poster@use.net
- [long] Mathematica 5.1 and memory: any garbage collection?