Re: 1GB Kernel Memory Limit/Out of memory errors
- To: mathgroup at smc.vnet.net
- Subject: [mg100621] Re: [mg100480] 1GB Kernel Memory Limit/[mg100582] Out of memory errors
- From: John Fultz <jfultz at wolfram.com>
- Date: Tue, 9 Jun 2009 03:58:00 -0400 (EDT)
- Reply-to: jfultz at wolfram.com
I was out of email contact for the weekend, and while going through the
weekend's MathGroup load, I noticed the following two similar posts,
from Friday night and Sunday night. The Friday one doesn't seem to have
had any public responders, so I thought I'd respond to both in one shot.
First, let me say categorically that Mathematica does not have a 1
gigabyte limit, and it has no limitation based solely on the amount
of physical RAM. Like virtually every other Windows application out
there, Mathematica does not manage its own paging file or its own memory
limitations. It merely allocates memory using Microsoft's implementation
of the C 'malloc' function, which is subject to the standard limitations
imposed by the operating system. On a Win32 system, these limitations
impose a per-process limitation of 2G of addressable space to the process,
or 3Gb if booted using the /3GB switch and if the application is compiled
appropriately to use the 3Gb, which Mathematica is. This addressable
space is backed by virtual memory, which is the collection of physical
RAM and page files managed directly by Windows itself.
Practically, you're never going to see anything very close to that
limitation being used by MemoryInUse. Some of the memory which is
addressable is not going to be directly available for use by Mathematica
for several reasons. But, in practice, I've been able to get a Mathematica
kernel to use around 1.85Gb (if memory serves me correctly...it's been
a while since I've done this) on a 2Gb system.
However, there are several legitimate reasons why a Mathematica kernel
could issue an out of memory message when MemoryInUse is displaying a
much lower number. Without seeing your programs, I could only speculate,
but here are some possibilities.
* Mathematica failed in the middle of making a very large memory
allocation because the memory just wasn't available. The math just
* Mathematica failed to make a smaller (but probably still in the tens
of megabytes) allocation because of memory fragmentation. Since there
are only 2Gb (or 3Gb) of addressable space, the pattern of past memory
allocations and frees may leave the currently allocated memory so
fragmented that no really large blocks of contiguous memory are left.
A single, 50Mb request for memory will fail if there are no contiguous
50Mb chunks of addressable space, regardless of how much memory is in use.
* Much less commonly, it's possible to exhaust other, much more limited,
memory-based resources. This includes stack space, which could affect
recursive calculations, and limitations on the number of handles, which
might come into play if accessing an extremely large number of files
without subsequently closing their streams.
A note about the memory fragmentation issue. If this were the problem,
then quite possibly you could run your evaluations on a 64-bit version
without any extra RAM or a larger paging file. The fact that a much
larger addressable space is available means that getting a contiguous
block of memory isn't a problem. But only the allocated blocks need to be backed by virtual memory, no matter what their addresses are.
jfultz at wolfram.com
User Interface Group
Wolfram Research, Inc.
On Fri, 5 Jun 2009 03:02:49 -0400 (EDT), Pillsy wrote:
> Hi, all,
> I'm currently working on a problem that requires a whole lot of memory
> to complete. Sometimes, depending on the specific parameters being
> used, I will receive the dreaded, "No more memory available," message.
> Doing my best to track memory usage with MemoryInUse, MaxMemoryUsed
>  and the Windows Task Manager, I find that the kernel quits when it
> hits a limit of around one gigabyte.
> I'm using Version 7.0 on 32-bit Windows XP without the /3GB switch
> active; neither switching to a 64 bit OS nor activating the /3GB
> switch are possibilities. However, I was under the distinct impression
> that even without the /3GB switch active I should have a per-process
> memory limit of 2GB, not 1GB. Is there some other reason the kernel
> might quit, due to either Mathematica settings or OS settings?
> As for modifying my program I've done obvious things like setting
> $HistoryLength 0, Clearing things that are no longer needed, and
> none of them have done the trick in all cases I've needed to deal
> with. I've also done an array of not-so-obvious things that haven't
> worked either.
> Any suggestions would be greatly appreciated.
On Mon, 8 Jun 2009 03:05:48 -0400 (EDT), Weldon MacDonald wrote:
> I'm operating on some largish problems in which I'm making a graph,
> using MakeGraph from the Combinatorica package. When I exceed a
> certain size, the thing runs for a bit and then gives an out of memory
> error and then the kernel shuts down. I'm running Vista on a dual core
> with 3Gb of RAM.
> I've tracked down the problem, I think. Mathematica runs the operation
> in RAM, when it exceeds the on board RAM, it shuts down. My question
> is why doesn't it build a pagefile like most windows applications?
> Then I could run anything that didn't actually use up the free disk
Prev by Date:
Documentation Methods and Efficiencies
Next by Date:
Re: performance // Re: Re: Why is recursion so
Previous by thread:
Re: Future for Mathematica on Solaris
Next by thread:
"Parallelize" and Symbolic computation