[long] Mathematica 5.1 and memory: any garbage collection?
- To: mathgroup at smc.vnet.net
- Subject: [mg59697] [long] Mathematica 5.1 and memory: any garbage collection?
- From: some.poster at use.net
- Date: Thu, 18 Aug 2005 00:16:30 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
Hi, I've been using Mathematica v5.1 under XP for some time now and found it largely improved over previous versions, at least for my problems. Only recently did I begin to handle some real data: rather large List objects. I have to routinely select objects within lists as large as, say, 46000 * 512 * 20 items, items themselves not being atomic. Of course I never expected to be able to use functionnal style for dealing with such large guys on a 1Gb RAM PC: an outer For loop hands one object (out of 46000) at a time, then functionnal style handling follows. In this process I create variable-sized intermediate objects that are either accumulated in decimated lists or discarded altogether. It turns out that my typical intermediate objects eat up to 2.5 Mb each. This sounds a little of a memory hog, but anyway, I can live with that. My problem lies with the fact that memory used at intermediate stage seems never be freed, even whith explicit Clear, Remove or any assignment to {} or . in whatever order. How can one recover the memory freed (if some is ever effectively freed)? Is there _any_ garbage collection in the guts of this gazillion dollars piece of code? I have no problem with the idea of regularly invoking some function to compact the heap (should the kernel be unable to perform it by itself). The question is: what's its name? How do you reclaim memory, you serious users who do computations on large objects with numerous intermediate values? At the end, I tried this, from a fresh kernel load: In[1]:= $ProductInformation Out[1]= {ProductIDName\[Rule]Mathematica, ProductKernelName\[Rule]Mathematica 5.1 Kernel, ProductVersion\[Rule]5.1 for Microsoft Windows (October 25, 2004), ProductVersionNumber\[Rule]5.1} In[2]:= MemoryInUse[] Out[2]= 2124904 In[3]:= (* let's create a moderate-size object *) try=Tuples[{a,b,c,d,e,f},7]; ByteCount[try] {MemoryInUse[],MaxMemoryUsed[]} Out[4]= 14556704 Out[5]= {16684640,16685792} In[6]:= (* simple mapped function *) res=Map[2*#&,try]; {MemoryInUse[],MaxMemoryUsed[]} Out[7]= {93949328,103186832} (* Ouch ! So much for a _simple_ test *) In[8]:= (* forgetting a large expression doesn't free _any_ memory *) try={}; {MemoryInUse[],MaxMemoryUsed[]} Out[9]= {93951056,103186832} In[10]:= (* no more success this way *) try=.; {MemoryInUse[],MaxMemoryUsed[]} Out[11]= {93952648,103186832} In[12]:= (* ... neither a sweep does *) Clear[try]; {MemoryInUse[],MaxMemoryUsed[]} Out[13]= {93954320,103186832} In[14]:= (* ... nor a complete wash *) Remove[try]; {MemoryInUse[],MaxMemoryUsed[]} Out[15]= {93955944,103186832} In[16]:= (* freeing the result also seems beyond possible *) res={}; {MemoryInUse[],MaxMemoryUsed[]} Out[17]= {93958112,103186832} In[18]:= (* still no luck, dad *) res=.; {MemoryInUse[],MaxMemoryUsed[]} Out[19]= {93958952,103186832} In[20]:= (* ahem, what _is_ Clear supposed to do? *) Clear[res]; {MemoryInUse[],MaxMemoryUsed[]} Out[21]= {93960448,103186832} In[22]:= (* ahem, what _is_ Remove supposed to do? *) Remove[res]; {MemoryInUse[],MaxMemoryUsed[]} Out[23]= {93961936,103186832} In[24]:= (* sharing common subexpressions somehow works, slow as hell *) (* please note: even after removing all values and symbols in all possible ways, try and res are _still_ somewhere in memory since Share _really_ does something! *) (* Share should share nothing as we don't have any object stored, safe perhaps symbols a..f *) (* when we use larger objects or more intermediate results like res we get the kernel to shutdown very quickly at this rate *) Share[] {MemoryInUse[],MaxMemoryUsed[]} Out[24]= 62824872 Out[25]= {31138184,103186832} (* did you expect to see about the same value as on the first call above? *)
- Follow-Ups:
- Re: [long] Mathematica 5.1 and memory: any garbage collection?
- From: ggroup@sarj.ca
- Re: [long] Mathematica 5.1 and memory: any garbage collection?
- From: Chris Chiasson <chris.chiasson@gmail.com>
- Re: [long] Mathematica 5.1 and memory: any garbage collection?
- From: Sseziwa Mukasa <mukasa@jeol.com>
- Re: [long] Mathematica 5.1 and memory: any garbage collection?
- From: stephen layland <layland@wolfram.com>
- Re: [long] Mathematica 5.1 and memory: any garbage collection?