Re: Re: Re: Re: "No more memory available" -- a recurring problem
- To: mathgroup at smc.vnet.net
- Subject: [mg68453] Re: Re: [mg68423] Re: [mg68388] Re: [mg68356] "No more memory available" -- a recurring problem
- From: "Charlie Brummitt" <cbrummitt at wisc.edu>
- Date: Sat, 5 Aug 2006 03:46:58 -0400 (EDT)
- References: <200608020924.FAA28575@smc.vnet.net> <D6C89613-F0EB-4D7E-9E29-2FFAA6FB40D4@yale.edu> <321730f50608021216o11d69435g5d278a6b7d06dfa5@mail.gmail.com> <200608031007.GAA15786@smc.vnet.net> <200608040759.DAA01108@smc.vnet.net> <4035DD91-055A-434E-A2F4-AC6F14A296CB@yale.edu> <321730f50608040955g5ca1e5bdq1c40d6f76aead70d@mail.gmail.com> <6ED37A1E-916F-4F36-9639-1632859206F4@yale.edu>
- Sender: owner-wri-mathgroup at wolfram.com
Eureka! I figured it out. In short, I had the NDSolve MaxSteps option set to Infinity, which was causing the "No more memory available" errors. By reducing this option to 20,000 and wrapping Check[] around the instances in which I use NDSolve, I seem to have solved the problem. A heartfelt thanks to all those who have helped me solve this problem -- János Lobb, Bill Rowe, James Gilmore, and Bruce Miller. Your thoughtful comments and persistence uplifted me from some dark days. I can't thank you enough. Now, finally, back to the research... Cheers, Charlie On 8/4/06, János <janos.lobb at yale.edu> wrote: > Charlie, > > I would set $RecirsionLimit and its cousins to Infinity. You might > also hit a built in limit of Mathematica that is not obvious and > might be processor dependent. /I managed to to that with GraphPlot > and the G5 duo Core :) / > > Are you able to run other Mathematica programs on the MacBook Pro > fine ? /Like creating a huge array that would eat up minimum half a > GB memory/. If yes, then you have to look your program with a fine > tooth comb and nail it down where does it fail. You might able to do > it with a group of Print statements placed at the right places in > your program. If you can nail down what particular statement is > offending the MacBook Pro, then you will have a good chance to fix it. > > If it dies in the middle of an NDsolve, then try that statement with > different parameters if you can or try to program it out with a > replacement/approximation. > > You might want to look the IMTEK Mathematica Supplement (IMS) > > http://www.imtek.de/simulation/ > > where they provide some other methods to solve PDE-s and use their > packages to see if those fail or not. > > Good luck, > > János > > > > > On Aug 4, 2006, at 12:55 PM, Charlie Brummitt wrote: > > > János, > > > > Below are my responses. > > > >> Do you have any Mathematica.crash.log or Mathkernel.crash.log file in > >> your ~Library/Logs/Crashreporter folder ? > >> If yes, does the datetime entries coincide with the kernel crashes ? > > > > I don't see any Mathematica crash logs in my > > ~Library/Logs/Crashreporter folder (only one file called > > mds.crash.log, which I doubt is Mathematica-related). > > > >> Did you try to eliminate or cut back on History /$HistoryLength /? > > > > I had set $HistoryLength to 5 all along. Since learning that > > $HistoryLength can take up a lot of memory, I set it to zero, but I > > still get the "No more memory available" message often. > > > >> Did you set any iteration related env variables Wolfram provide to > >> Infinity like $IterationLimit and $RecusionLimit, etc.... > > > > No, I did not change any of those from the default. I rarely, if ever, > > get other error messages, such as "reached the recursion limit," > > preceding the "No more memory available" message. > > > > Thanks again for all your help, János. > > > > Charlie > > > > > >> On Aug 4, 2006, at 3:59 AM, Charlie Brummitt wrote: > >> > >> > I took my MacBook Pro into the university computer support & repair > >> > office, and they concluded that the RAM is fine and that it > >> probably > >> > is not a hardware problem. I ran memtest (www.memtestosx.org/) > >> on the > >> > new memory stick alone (it's from OMNI Technology) and found no > >> > errors. > >> > > >> > That leaves me with three possibilities: > >> > 1) my code can simply be too difficult for a personal computer and > >> > some sort of grid or supercomputer is required > >> > 2) there is a bug in Mathematica, in the act of porting > >> Mathematica to > >> > Intel (i.e. to being universal), or in OSX which causes memory > >> > failures even when there is plenty of memory and virtual memory > >> > available > >> > 3) my code is poorly written and needs radical change. To answer > >> Bill > >> > Rowe's question, I would consider myself an intermediate > >> Mathematica > >> > user, and I think I'm fairly well-versed in the area of > >> Mathematica in > >> > which I'm working (namely, solving and analyzing differential > >> > equations). > >> > > >> > Basically, my calculation consists of using NDSolve thousands of > >> times > >> > to solve and evaluate PDE's in order to find ones that are chaotic. > >> > > >> > I'm not sure what else to do. Contact Wolfram maybe? > >> > > >> > I guess this ends my research project. > >> > > >> > Thanks to János and Bill for their helpful comments. > >> > > >> > Charlie > >> > > >> > On 8/3/06, János <janos.lobb at yale.edu> wrote: > >> >> Charlie, > >> >> > >> >> Your MacBook Pro has an Intel core duo processor and looks to me > >> >> that Mathematica is using just one of the cores. I do not know > >> what > >> >> You can do about it. Probably nothing. If you can put your > >> hand on > >> >> a Personal Grid edition that would engage the second core more > >> >> vigorously. > >> >> > >> >> I do not see that you have a memory problem as regarding the > >> usage of > >> >> the memory reported by top. > >> >> > >> >> You might have a bad memory chip and that is crashing your > >> >> program. Unfortunately we see bad memory coming even from Apple > >> >> these days with newer Macs. There is a service CD that came with > >> >> your MacBook Pro. Try to run the memory test from it all night > >> in a > >> >> loop and see if it is caching anything. > >> >> > >> >> You might also want to do a hard drive check with Diskwarrior. > >> Wooly > >> >> disks can lead to wooly virtual memory and causing crashes. > >> >> > >> >> Look if you have anything in ~/Library/Logs/Crashreporter > >> regarding > >> >> Mathematica.crash.log or Mathkernel.cash.log. /I have plenty in > >> >> mine for both :)/ If you find any, send them to > >> support at wolfram.com > >> >> and ask their opinion. > >> >> > >> >> I also recommend to do additional exhausting tests for the > >> >> motherboard and other elements of the machine by using the service > >> >> CD. > >> >> > >> >> In summary it looks like a hardware problem - unless you have > >> other > >> >> long running Mathematica programs which run just fine. > >> >> > >> >> With the best, > >> >> > >> >> János > >> >> > >> >> On Aug 2, 2006, at 3:16 PM, Charlie Brummitt wrote: > >> >> > >> >>> János, > >> >>> > >> >>> Thanks for the advice. I'm not really sure how to interpret the > >> >>> information in the terminal window, so I have attached > >> >>> screenshots of > >> >>> it at various stages (see the names of the images). I have > >> attached > >> >>> them in chronological order. > >> >>> > >> >>> It seems to me that Mathematica is only being allocated 50% of > >> >>> the CPU > >> >>> and 100 megabytes of RAM! When I did the "top" on my 12" > >> PowerBook, > >> >>> 93% of the CPU was being used while Mathematica was running my > >> code. > >> >>> How do I change this on my 15" MacBook Pro? > >> >>> > >> >>> My 12" PowerBook is able to run this code overnight and very > >> rarely > >> >>> gives the "ran out of memory" error message. I would like to > >> be able > >> >>> to run my code overnight on my MacBook Pro, but it can only run > >> >>> for 5 > >> >>> to 10 minutes before it runs out of memory and completely halts. > >> >>> Thus > >> >>> I can't "set it and leave it"; instead, I must always be at my > >> >>> computer to restart the calculation every 5 to 10 minutes. Any > >> >>> suggestions? > >> >>> > >> >>> > >> >>> Thanks for all your help, János. > >> >>> > >> >>> Charlie > >> >>> > >> >>> On 8/2/06, János <janos.lobb at yale.edu> wrote: > >> >>>> > >> >>>> On Aug 2, 2006, at 5:24 AM, Charlie Brummitt wrote: > >> >>>> > >> >>>>> I keep getting the following error message while running a > >> search > >> >>>>> of many PDE's: > >> >>>>> > >> >>>>> "No more memory available. > >> >>>>> Mathematica kernel has shut down. > >> >>>>> Try quitting other applications and then retry." > >> >>>>> > >> >>>>> This message is becoming so common that it is crippling my > >> >>>>> research > >> >>>>> project, which is to find the simplest PDE with one quadratic > >> >>>>> nonlinearity that is chaotic. > >> >>>>> > >> >>>>> I have Googled and searched the Mathgroup archives for help, > >> and I > >> >>>>> employed the following fixes: > >> >>>>> > >> >>>>> --CODE & SOFTWARE-- > >> >>>>> 1) Share[] -- does not help because my code rarely has common > >> >>>> elements > >> >>>>> that Share[] could consolidate. > >> >>>>> > >> >>>>> 2) I used Module[] and made as many variables local as > >> possible. I > >> >>>>> eliminated extra variables and functions in addition to adding > >> >>>> Clear[] > >> >>>>> in several places in the code to clear variables that are no > >> >>>>> longer > >> >>>>> needed. > >> >>>>> > >> >>>>> 3) I streamlined and optimized the code in general, and I > >> made the > >> >>>>> routines as simple and least data-intensive as possible. > >> >>>>> > >> >>>>> 4) I never have any other applications open when running > >> >>>> Mathematica. > >> >>>>> (Unfortunately, closing other applications is the only > >> >>>> suggestion that > >> >>>>> Mathematica provides in the "out of memory" error message.) > >> >>>>> > >> >>>>> 5) I should note that I run the latest version of Mathematica > >> >>>>> (5.2.2.0). > >> >>>>> > >> >>>>> --HARDWARE-- > >> >>>>> 1) I upgraded from a 12" PowerBook to a 15" MacBook Pro. This > >> >>>> computer > >> >>>>> has an 2.16 GHz Intel Core Duo processor. > >> >>>>> > >> >>>>> 2) I upgraded the RAM on this new computer from 1 GB to 2 GB. > >> >>>>> > >> >>>>> 3) I now have much more free hard drive space (20 GB) in case > >> >>>>> Mathematica needs to use virtual memory. In addition, this hard > >> >>>> drive > >> >>>>> is 7200 rpm versus the standard 5400 rpm. > >> >>>>> > >> >>>>> > >> >>>>> I have been using functions like MemoryInUse[] and On > >> >>>> [MemoryConserve] > >> >>>>> to monitor the use of memory. I almost always find that I am > >> only > >> >>>>> using a few megabytes at a time (usually 5-10 MB, sometimes as > >> >>>> high as > >> >>>>> 100 or once 1000 after a large computation). However, despite > >> >>>> finding > >> >>>>> that I usually use only a few megabytes during my > >> computations, I > >> >>>>> often get the above "out of memory" error message, and I can > >> never > >> >>>>> figure out why and how much memory was needed in that > >> particular > >> >>>>> computation. > >> >>>>> > >> >>>>> > >> >>>>> Some questions: > >> >>>>> 1) How can I tell whether Mathematica is using virtual > >> memory or > >> >>>> not? > >> >>>>> It seems to me that it is not. I have 20 GB of hard drive space > >> >>>> free > >> >>>>> and it never seems to use it. Why should I ever run out of > >> >>>> memory if I > >> >>>>> have 20 GB free in case it needs more memory than my RAM > >> provides? > >> >>>>> > >> >>>>> 2) Is there a way to determine how much of my 2 GB of RAM are > >> >>>>> allocated to Mathematica? Can I increase how much RAM > >> >>>> Mathematica can > >> >>>>> access? > >> >>>>> > >> >>>>> Thanks in advance for any help you may be able to provide! I > >> >>>> greatly > >> >>>>> appreciate it. If you need more information on my code or the > >> >>>>> computations I am doing, I would be happy to provide it. > >> >>>>> > >> >>>>> Regards, > >> >>>>> Charlie > >> >>>> > >> >>>> If you open a Terminal window on the side and type "top" and hit > >> >>>> return and start up your application, what do You see for real > >> >>>> memory > >> >>>> usage for the kernel and the front end from top? > >> >>>> > >> >>>> János > >> >>>> <before mathematica.jpeg> > >> >>>> <mma loaded, code loaded, pre-execution.jpeg> > >> >>>> <running1.jpeg> > >> >>>> <running2.jpeg> > >> >>>> <running3.jpeg> > >> >>>> <running4.jpeg> > >> >>>> <ran out of memory.jpeg> > >> >> > >> >> > >> > >> > > > > ---------------------------------------------- > Trying to argue with a politician is like lifting up the head of a > corpse. > (S. Lem: His Master Voice) > >
- References:
- "No more memory available" -- a recurring problem
- From: "Charlie Brummitt" <cbrummitt@wisc.edu>
- Re: "No more memory available" -- a recurring problem
- From: János <janos.lobb@yale.edu>
- Re: Re: "No more memory available" -- a recurring problem
- From: "Charlie Brummitt" <cbrummitt@wisc.edu>
- "No more memory available" -- a recurring problem