Re: Memory restoring form within Mathematica ?
- To: mathgroup at smc.vnet.net
- Subject: [mg34088] Re: [mg34073] Memory restoring form within Mathematica ?
- From: Sseziwa Mukasa <mukasa at jeol.com>
- Date: Thu, 2 May 2002 03:49:31 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
On Wednesday, May 1, 2002, at 08:00 AM, Justus Heimann wrote: > > Hi, > > I got a question concerning the memory handling in Mathematica 4.0 > Windows. > > Within an engineering optimization application I have to deal with very > big numerical matrices simultaneously in a Mathematica session. In order > to not overflow the already large physical memory (1.5 GB!) I arranged > to save and reload (when needed) some of these matrices from and to disk > during a session. > > But I found that, e.g., the Mathematica functions Import/Export or > ReadListBinary/WriteBinary (binary.exe) alone use very much physical > memory, in addition to the memory consumption of the matrices itselves, > to do the job. And even though the memory consumption of the Mathematica > kernel, as indicated by MemoryInUse[], rather seems to be maintainable, > at the same time the physical memory used on the Windows system level, > as indicated by the Windows Task-Manager, is much higher or even > overfloating ?! What is the reason for this discrepancy and how to > overcome this ? > > In order to clear the physical Windows memory I tried: > ClearAll[matrix1, matrix2, ...]; Unprotect[Out]; Clear[Out]; > Protect[Out]; > or > ClearAll[matrix1, matrix2, ...]; ClearInOut[]; > of the CleanSlate Package. That way the memory used by the Mathematica > kernel is restored correspondigly, but, unfortunately, the memory > consumption at the Windows system level is unaffected ???! > > So is their a way to successfully restore Mathematica memory and at the > same time the physical Windows memory from within a Mathematica session > without killing the session resp. the kernel ? > > Many thanks > Justus Heimann > > I am not a Windows internals expert but I suspect you are seeing a side effect of decisions made by the Window's virtual memory management system and Mathematica kernel. Basically, if an application requests memory and Windows fulfills that request, that memory is considered as belonging to that application until the application explicitly returns that memory to the OS. For performance reasons Mathematica may assume that if you requested a large block of memory you may have need of it in the future so it doesn't return memory allocated from the OS until the kernel quits. This is not an unreasonable policy since it is likely that a calculation that uses a lot of memory will be executed more than once. Another possibility is the kernel will hold on to pages for some length of time before returning them to the system and you are simply seeing the situation before the kernel has returned the memory. At any rate the number returned from MemoryInUse however is the amount of the allocated space the kernel is actually using currently not the size of all memory allocated from the OS, hence the apparent mismatch. In a good virtual memory subsystem (feel free to start a religious debate about the quality of various VMs here) it isn't really an issue when an application requests a large number of unused memory pages because the paging system will swap the unused pages to disk anyway freeing up faster real memory for applications that actually are accessing memory. Regards, Sseziwa