Re: Preventing swapping (high memory use)

*To*: mathgroup at smc.vnet.net*Subject*: [mg99861] Re: [mg99860] Preventing swapping (high memory use)*From*: Leonid Shifrin <lshifr at gmail.com>*Date*: Sat, 16 May 2009 18:24:27 -0400 (EDT)*References*: <200905160922.FAA26903@smc.vnet.net>

Hi Szabolcs, How about this: In[1] = ClearAll[totalMemoryConstrained]; SetAttributes[totalMemoryConstrained, HoldRest]; Module[{memException}, totalMemoryConstrained[max_, body_, failexpr_] := Catch[MemoryConstrained[body, Evaluate[ If[# < 0, Throw[failexpr, memException], #] &@(max - MemoryInUse[])], failexpr], memException]]; To test, I start with a fresh kernel: In[2] = MemoryInUse[] Out[2] = 5815968 In[3] = n = 0; lst = {}; totalMemoryConstrained[10000000, For[lst = {}; n = 10000, n < 100000, n++, lst = Join[lst, Range[n]]], Print[n]] Out[3] = 10043 (Printed) In[4] = MemoryInUse[] Out[4] = 8257328 I deliberately use globals to prevent them from being garbage-collected after the function returns. Now take a limit smaller than the current usage: In[5] = n = 0; lst = {}; totalMemoryConstrained[3000000, For[lst = {}; n = 10000, n < 100000, n++, lst = Join[lst, Range[n]]], Print[n]] Out[3] = 0 (Printed) - we just stop right away (through exception, obviously). This of course relies on how often does MemoryConstrained[] recompute its parameters, and how precise is MemoryInUse[]. My simple tests indicated that it works, up to plus-minus a few Mb, but obviously more tests are needed. Regards, Leonid 2009/5/16 Szabolcs Horv=E1t <szhorvat at gmail.com> > > Sometimes I don't realize how much memory a computation would take up > (or I simply make a mistake, and start a calculation that requires too > much memory), and my computer starts swapping. > > This makes Mathematica (and often the OS too) completely unresponsive > for a long time. In this case my only hope is that in spite of the > system unresponsiveness I can manage to kill the kernel, so that I can > at least save the notebook ... > > Is there any way to prevent swapping, for example by putting a cap on > the kernel's memory use? Even if the kernel just quits when it runs out > of memory, I still have the notebook. But if the computer gets so > unresponsive that I have to restart it, then I lose the notebook too, > which is a much more serious loss ... > > I tried using MemoryConstrained (with $Pre), but it seems to constrain > only the memory used by current computation, not the full kernel memory > use. > > > To summarize: I'm simply looking for a way to prevent the computer from > locking up because of excessive swapping. >

**References**:**Preventing swapping (high memory use)***From:*Szabolcs Horvát <szhorvat@gmail.com>