Re: Re: negative FileByteCount
- To: mathgroup at smc.vnet.net
- Subject: [mg83273] Re: [mg83201] Re: negative FileByteCount
- From: "Jean-Marc Gulliet" <jeanmarc.gulliet at gmail.com>
- Date: Fri, 16 Nov 2007 05:39:29 -0500 (EST)
- References: <200711151030.FAA08772@smc.vnet.net>
On Nov 15, 2007 9:05 PM, John Fultz <jfultz at wolfram.com> wrote: > A fairly informative post except for one critical point which is in error... > > > Also, applications running concurrently must share this 2GB space > > This is not true, although, in fairness to you, the Wikipedia article certainly > implies it is. It's pretty easy to verify that it's not true using two copies > of the kernel, a sufficiently large page file, a couple of huge Range[] > commands, and a whole lot of time to wait for the swapping. Oops! Thank you John for having caught this silly mistake of mine. > Raymond Chen, one of the Microsoft techies who has been around for a truly long > time and seen it all, wrote a very enlightening series of articles on his blog > about the various issues and myths of how and why Windows 32-bit is limited the > way it is. The article dealing with the specific myth reiterated here is... > > http://blogs.msdn.com/oldnewthing/archive/2004/08/09/211356.aspx > > The entire series of articles is indexed here... > > http://blogs.msdn.com/oldnewthing/archive/2004/08/22/218527.aspx Very interesting blog, indeed. Best regards, Jean-Marc > Another point you made is dead on, and I'd like to highlight it because it's the > source of many "why can't I get access to all of my memory" questions... > > > some applications (or at least specific functions of these applications) > > may require continuous block of memory > > Mathematica requires contiguous blocks of memory for very large packed arrays > (that the memory is contiguous is part of what makes packed arrays so efficient, > so this really is a feature). The memory usage of Mathematica begins fragmented > as a result of where certain system DLLs decide to load themselves in the > virtual address range. And, to be honest, there's a bit of fragmentation that's > caused by our own loading of code in less than optimal locations (specific to > Windows...we're looking into this for a future release). But fragmentation is > often going to be induced by the history of memory usage during the session. > > If you anticipate using large blocks of memory on a 32-bit system, it's best to > allocate them as early in the session as possible to avoid this fragmentation. > But it really is better to get a 64-bit system. Such systems extremely > affordable these days (although some vendors make it very hard or confusing to > get a 64-bit version of Windows). 64-bit Mathematica runs very well. Many > people inside Wolfram, myself included, run it daily. > > And, incidentally, since this thread started off dealing with large files, > rather than large memory allocations...yes, it ought to be the case (but isn't) > that 32-bit versions should be able to deal with very large files. Even though > you couldn't expect to load files in their entirety, it's certainly reasonable > to want to call SetStreamPosition and load pieces of them. It's a limitation of > Mathematica, and one that affects both the front end and kernel, although it > really isn't a very easy problem to fix. Many 32-bit applications suffer from > similar issues, and it requires a lot of very intentional and difficult > (particularly for large applications) engineering to overcome it. > > I've often seen "support of very large files" (meaning, of course, >2G) touted > as a major feature (rather than a minor bug fix) on the short list of features > for new versions of various industrial applications. > > Sincerely, > > John Fultz > jfultz at wolfram.com > User Interface Group > Wolfram Research, Inc. > > > > On Thu, 15 Nov 2007 05:30:07 -0500 (EST), Jean-Marc Gulliet wrote: > > [My reply is at the bottom of this email.] > > > > rych wrote: > > > >> Thanks for response > >> Mathematica 6.0.1.0 > >> Windows XP Pro SP2 32 bit > >> 2GB of RAM, Commit charge: 2113/5977M. > >> Perhaps, I should increase the page file, say to 2x3GB? It's if the > >> FileByteCount actually loads the whole file into memory to find out > >> about its size. I'll report later whether it worked. Ultimately, I'd > >> like to know the size limit and how to load such big files by parts. > >> Igor > >> > >> On 10 Nov, 08:51, Jean-Marc Gulliet <jeanmarc.gull... at gmail.com> > >> wrote: > >>> rych wrote: > >>>> Is there a maximum file size permitted in Mathematica? Because it > >>>> seems I can't work with my 3GB file, > >>> <snip> > >>> > >>> It all depends on what version of Mathematica you are using (32 or 64 > >>> bits), your operating system, the available memory you have when > >>> importing the file, possibly the type of data imported... > >>> > >>> Thus the following questions: What version of Mathematica do you use? > >>> What platform (hardware and operating system do you use? How much > >>> memory > >>> is available for Mathematica (or are you running many applications > >>> simultaneously or just Mathematica)? > >>> > > Summary: if Mathematica, or any other applications, must load 3GB of > > data into memory, this is doomed to failed on a 32-bit system. > > > > The address space of any 32-bit system, regardless of its operating > > system (Windows, Linux, MacOs, Solaris, UNIX, etc.) or chip maker > > (Intel, AMD, PowerPC, etc.), is limited to 4GB (i.e. the smallest > > address is 0 and the largest is (2^32)-1 = 4,294,967,295). This limit is > > due to the size of the address bus (32-bit width), bus that allows the > > exchange of data between the processor and the main memory. > > > > On Windows, the main memory is managed as virtual memory, i.e. the main > > memory is made of physical ram and a page file. Note that the > > addressable virtual memory cannot exceeds 4GB even though the page file > > may be created as large as you wish. > > > > In addition of this limitation to a maximum of 4GB of addressable > > memory, the address space is variously divided between the operating > > system and the user(s). Still on Windows XP, the kernel (system memory) > > gets 2GB and the applications get the other 2GB. (It is possible on > > certain circumstances to increase this ratio in favor of the user > > applications up to 3GB or so.) > > > > Also, applications running concurrently must share this 2GB space and > > some applications (or at least specific functions of these applications) > > may require continuous block of memory (similarly to a hard disk, the > > main memory will become fragmented during a work session). > > > > Conclusion: if Mathematica (or any other applications) must load 3GB of > > data into memory, this is doomed to failed on a 32-bit system. > > > > The following pages may be particularly worth reading: > > > > "Windows XP Professional x64 Edition, From Wikipedia, the free > > encyclopedia" > > > > http://en.wikipedia.org/wiki/Windows_XP_Professional_x64_Edition > > > > particularly the first section is of uttermost importance regarding > > memory differences between 32- and 64-bit architectures. > > > > "RAM, Virtual Memory, Pagefile and all that stuff" > > http://support.microsoft.com/default.aspx/kb/555223 > > > > Regards,
- References:
- Re: negative FileByteCount
- From: Jean-Marc Gulliet <jeanmarc.gulliet@gmail.com>
- Re: negative FileByteCount