Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Re: Preventing swapping (high memory use)

• To: mathgroup at smc.vnet.net
• Subject: [mg99861] Re: [mg99860] Preventing swapping (high memory use)
• From: Leonid Shifrin <lshifr at gmail.com>
• Date: Sat, 16 May 2009 18:24:27 -0400 (EDT)
• References: <200905160922.FAA26903@smc.vnet.net>

Hi Szabolcs,

How about this:

In[1] =

ClearAll[totalMemoryConstrained];
SetAttributes[totalMemoryConstrained, HoldRest];
Module[{memException},
  totalMemoryConstrained[max_, body_, failexpr_] :=
   Catch[MemoryConstrained[body,
     Evaluate[
      If[# < 0, Throw[failexpr, memException], #] &@(max -
         MemoryInUse[])], failexpr], memException]];

To test, I start with a fresh kernel:

In[2] = MemoryInUse[]

Out[2] = 5815968

In[3]  =

n = 0;
lst = {};
totalMemoryConstrained[10000000,
 For[lst = {}; n = 10000, n < 100000, n++,
  lst = Join[lst, Range[n]]], Print[n]]

 Out[3] = 10043 (Printed)

In[4] = MemoryInUse[]

Out[4] = 8257328

I deliberately use globals to prevent them from being garbage-collected
after the function returns.

Now take a limit smaller than the current usage:

In[5]  =
n = 0;
lst = {};
totalMemoryConstrained[3000000,
 For[lst = {}; n = 10000, n < 100000, n++,
  lst = Join[lst, Range[n]]], Print[n]]

Out[3] = 0 (Printed)

 - we just stop right away (through exception, obviously).

This of course relies on how often does MemoryConstrained[]
recompute its parameters, and how precise is MemoryInUse[].
My simple tests indicated that it works, up to plus-minus a few
Mb, but obviously more tests are needed.


Regards,
Leonid


2009/5/16 Szabolcs Horv=E1t <szhorvat at gmail.com>

>
> Sometimes I don't realize how much memory a computation would take up
> (or I simply make a mistake, and start a calculation that requires too
> much memory), and my computer starts swapping.
>
> This makes Mathematica (and often the OS too) completely unresponsive
> for a long time.  In this case my only hope is that in spite of the
> system unresponsiveness I can manage to kill the kernel, so that I can
> at least save the notebook ...
>
> Is there any way to prevent swapping, for example by putting a cap on
> the kernel's memory use?  Even if the kernel just quits when it runs out
> of memory, I still have the notebook.  But if the computer gets so
> unresponsive that I have to restart it, then I lose the notebook too,
> which is a much more serious loss ...
>
> I tried using MemoryConstrained (with $Pre), but it seems to constrain
> only the memory used by current computation, not the full kernel memory
> use.
>
>
> To summarize:  I'm simply looking for a way to prevent the computer from
> locking up because of excessive swapping.
>

• References:
  • Preventing swapping (high memory use)
    • From: Szabolcs Horvát <szhorvat@gmail.com>