Re: Does MemoryConstrained count stack space?
- To: mathgroup at smc.vnet.net
- Subject: [mg117444] Re: Does MemoryConstrained count stack space?
- From: Leonid Shifrin <lshifr at gmail.com>
- Date: Sat, 19 Mar 2011 05:16:38 -0500 (EST)
Alexey,
Interesting. I will have to give it more thought. But here is my guess (may
be, this is a complete nonsense). Mathematica probably maps its evaluation
stack onto the stack maintained by the C run-time, or, OS stack
space (this is also mentioned in the documentation). Most things Mathematica
creates, such as lists (Table[Random[],...], or whatever) are immutable and
can not be modified, but
only copied. These are temporarily created in dynamically allocated RAM (the
heap), which is
distinct from the stack space. Since the stack space is a limited resource,
my guess would be that for the
expressions we see on the stack, only pointers to these expressions are
really kept there, while expressions
themselves are kept on the heap. If this is indeed true, then in your
example, the real thing that is kept on
the stack for your Plus@@Table is an expression where the pointer is used
instead of the full copy of a
huge array (this is anyway how things are done in C - no one in their right
mind would copy a full huge array
onto the stack, this is also grossly inefficient in terms of run-time, not
just memory). So, I would guess that
most of the memory consumption you are observing is from the heap memory,
not stack memory.
I would be very interested in knowing whether or not my guess is correct.
Regards,
Leonid
On Fri, Mar 18, 2011 at 11:18 AM, Alexey Popkov <lehin.p at gmail.com> wrote:
> Leonid,
>
> As an illustration for the problem, try the following:
> In[1]:= MaxMemoryUsed[]
> MemoryConstrained[
> Quiet@Module[{f, n = 0},
> f[x_] := (n++; f[x + 1] + Plus @@ Table[Random[], {10^7}]);
> Block[{$RecursionLimit = 3000}, f[0]]];,
> 90000000];
> MaxMemoryUsed[]
> Out[1]= 10071968
> Out[3]= 290681632
>
> You can see that MathKernel takes more than 267 Mb additional memory in
> peak
> although only less than 90 Mb additional memory was allowed by
> MemoryConstrained. This behavior is machine-dependent. Try to play with the
> memory constrain if you cannot reproduce such behavior with the original
> code.
>
> My experience shows that in practice this value may be 900 Mb and more!
> This
> often happens in complicated computations with heavy usage such functions
> as
> NDSolve and NIntegrate.
>
> P.S. I currently use Mathematica 7.0.1 under Windows 2000.
>
> Alexey
>
> ----- Original Message -----
> From: "Leonid Shifrin" <lshifr at gmail.com>
> To: "Alexey" <lehin.p at gmail.com>; <mathgroup at smc.vnet.net>
> Sent: Friday, March 18, 2011 4:05 AM
> Subject: Re: [mg117411] Does MemoryConstrained count stack space?
>
>
> > Alexey,
> >
> > My impression of the memory consumption by the stack has been that it is
> > (usually?)
> > much smaller than memory typically used for computations. In other words,
> > you have
> > just a relatively small amount of memory allowed to be used by the stack
> > until the kernel
> > crashes:
> >
> > In[1]:= MaxMemoryUsed[]
> >
> > Out[1]= 18009584
> >
> > In[2]:= Quiet@Module[{f, n = 0}, f[x_] := (n += 1; f[x + 1]);
> > TimeConstrained[Block[{$RecursionLimit = 65000}, f[0]], 0.5, n]];
> >
> > In[3]:= MaxMemoryUsed[]
> >
> > Out[3]= 29857584
> >
> > It is about 10 Mb only, and on my system (AMD Phenom X6, Win7, M8.0, 8Gb
> > RAM) setting the
> > $RecursionLimit to a slightly larger value results in a crash for this
> > particular function. Given that
> > generally the Mathematica's memory-constraining functions are working
> > approximately at best,
> > and given also that the stack space seems to often be only a small
> fraction
> > of the total memory
> > used in a computation, the memory-based method of avoiding crashes does
> not
> > seem promising
> > to me.
> >
> > If it is the crashes that you want to prevent, I'd set a fixed finite
> > $RecursionLimit, or, even better,
> > look carefully at your functions to see why they need such a deep
> recursion
> > and whether you can
> > rewrite the code to use iterations. OTOH, if you want to prevent the
> large
> > memory consumption,
> > evaluation stack seems to not be the main contributing factor. If your
> goal
> > is to prevent excessive
> > swapping, in this thread
> >
> >
>
> http://groups.google.com/group/comp.soft-sys.math.mathematica/browse_thread/thread/bc41b4a5f51fbcb8
> >
> > a function is developed which you may find useful.
> >
> > Regards,
> > Leonid
> >
> >
> > On Thu, Mar 17, 2011 at 2:34 PM, Alexey <lehin.p at gmail.com> wrote:
> >
> > > Hello,
> > >
> > > On the Documentation page for $RecursionLimit we read:
> > >
> > > "MemoryInUse and related functions do not count stack space. "
> > >
> > > And also:
> > >
> > > "On most computers, each level of recursion uses a certain amount of
> > > stack space. $RecursionLimit allows you to control the amount of stack
> > > space that Mathematica can use from within Mathematica. On some
> > > computer systems, your whole Mathematica session may crash if you
> > > allow it to use more stack space than the computer system allows. "
> > >
> > > Does MemoryConstrained count stack space? If not, is there another
> > > function that fulfills the following statement taken from the
> > > documentation page for MemoryConstrained:
> > >
> > > "MemoryConstrained generates an interrupt to stop the evaluation of
> > > expr if the amount of additional memory requested during the
> > > evaluation of expr exceeds b bytes."
> > >
> > > ?
> > >
> > > I mean a function that will count ALL additional memory requested
> > > during evaluation, including stack space.
> > >
> > >
> >
>