Re: Re: Excessive Mathematica memory use, revisited.
- To: mathgroup at smc.vnet.net
- Subject: [mg46919] Re: [mg46873] Re: Excessive Mathematica memory use, revisited.
- From: János <janos.lobb at yale.edu>
- Date: Tue, 16 Mar 2004 02:37:22 -0500 (EST)
- References: <c2rnjr$p13$1@smc.vnet.net> <200403130439.XAA15057@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
Another solution could be if Mathematica could use other computers' memory on the local - or wide - network without having Mathematica - or the kernel - installed on them. Of course it would need some help from the OS, but it is possible that on some OSs it is already there. I could imagine it in the following way. If I look top on my machine I see that 98M is wired, 120M is active, 176M is inactive, 394M is used and 1.6G is free. I can imagine that from the "free" the OS would designate just one third as "localfree" and the other two third as "networkfree". Other machines could tap into this "networkfree" reserve for their own purposes. On fast networks - Gbit and above - the allocation and synchronization of this network mapped memory could be as fast as disk based virtual memory or faster. János On Mar 12, 2004, at 11:39 PM, Jens-Peer Kuska wrote: > Hi, > > if you have a 16 bit image that need 500 MByte, Mathematica will > store the 16 bit number as 32 bit integers and the Mathematica kernel > needs 1 GByte. In allmost all cases Mathematica will try to > make a copy of the data when a function is called and a single copy > will overflow your physical memory. > > The only solution is to write a C/C++ MathLink program that store > the data as 16 bit shorts and add some C functions to manipulate > the data. > > I have used this technique several times for huge volume data and > Mathematica serve only as a comfortable command interface. > > Or can you split the image in smaller sections ?? > > Regards > Jens > > > "Virgilio, Vincent" wrote: >> >> Hello, >> >> First, thanks to Jens-Peer Kuska for responding to this issue back in >> September, >> >> http://forums.wolfram.com/mathgroup/archive/2003/Sep/msg00125.html. >> >> I am now working with a much more powerful machine: P4 2.6 GHz, 2GB >> RAM, >> Windows XP, Mathematica 5.0.0. >> >> My goal is still to manipulate very large images. Currently I am >> trying >> to use Experimental`BinaryImport to import an ~ 500MB image into >> Mathematica. Each sample is an unsigned 16-bit integer. >> >> I start the Windows Task Manager, display the process list, and sort >> by >> memory usage. >> >> Then, after issuing the following commands to a cold kernel, I watch >> the >> Mathematica kernel rise to the top of the above list, consume chunks >> of >> memory in steps of maybe 20MB, and exhaust physical memory. >> >> <<ImageProcessing` >> >> $MessagePrePrint=Short; >> >> $HistoryLength=0; >> >> <<Utilities`MemoryConserve` >> >> Off[MemoryConserve::start];Off[MemoryConserve::end] >> >> <<Experimental` >> >> image=ToPackedArray[BinaryImport["500MBimage",{"UnsignedInteger16"...} >> ]] >> ; >> >> There is memory exhaustion with or without ToPackedArray[]. >> >> I could understand this behaviour if Mathematica is storing each >> sample >> in a node of some sort of tree. Then the data volume would be >> multiplied >> by some factor determined by the tree node size. If this is the case, >> is >> there a way to avoid this, during import, with packed arrays? >> >> Is there anything obviously wrong or inefficient in the above code? >> I've >> tried to incorporte Jens' suggestions. One of his was to use Hold[] >> wherever possible; I can't see where it would be productive to do that >> here. >> >> Note, I see similar behaviour on a smaller scale with much smaller >> images (14MB, per earlier mail). Which is not a problem since the >> memory >> ceiling has gone up considerably. >> >> Perhaps 2GB is simply inadequate for this task, since I don't expect >> Mathematica to use a data tile cache. Other products do, but then they >> target a much more specific application domain. >> >> Regards and thanks, >> >> Vince Virgilio >> >> ************************************ >> This email and any files transmitted with it are proprietary and >> intended solely for the use of the individual or entity to whom they >> are addressed. If you have received this email >> in error please notify the sender. Please note that any views or >> opinions presented in this email are solely those of the author and >> do not necessarily represent those of ITT Industries, Inc. >> The recipient should check this email and any attachments for the >> presence of viruses. ITT Industries accepts no liability for any >> damage caused by any virus transmitted by this >> email. >> ************************************ > > ------------------------------------------------- clear perl code is better than unclear awk code; but NOTHING comes close to unclear perl code http://www.faqs.org/faqs/computer-lang/awk/faq/
- References:
- Re: Excessive Mathematica memory use, revisited.
- From: Jens-Peer Kuska <kuska@informatik.uni-leipzig.de>
- Re: Excessive Mathematica memory use, revisited.