MathGroup Archive 2007

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Re: negative FileByteCount

  • To: mathgroup at smc.vnet.net
  • Subject: [mg83273] Re: [mg83201] Re: negative FileByteCount
  • From: "Jean-Marc Gulliet" <jeanmarc.gulliet at gmail.com>
  • Date: Fri, 16 Nov 2007 05:39:29 -0500 (EST)
  • References: <200711151030.FAA08772@smc.vnet.net>

On Nov 15, 2007 9:05 PM, John Fultz <jfultz at wolfram.com> wrote:

> A fairly informative post except for one critical point which is in error...
>
> > Also, applications running concurrently must share this 2GB space
>
> This is not true, although, in fairness to you, the Wikipedia article certainly
> implies it is.  It's pretty easy to verify that it's not true using two copies
> of the kernel, a sufficiently large page file, a couple of huge Range[]
> commands, and a whole lot of time to wait for the swapping.

Oops! Thank you John for having caught this silly mistake of mine.

> Raymond Chen, one of the Microsoft techies who has been around for a truly long
> time and seen it all, wrote a very enlightening series of articles on his blog
> about the various issues and myths of how and why Windows 32-bit is limited the
> way it is.  The article dealing with the specific myth reiterated here is...
>
> http://blogs.msdn.com/oldnewthing/archive/2004/08/09/211356.aspx
>
> The entire series of articles is indexed here...
>
> http://blogs.msdn.com/oldnewthing/archive/2004/08/22/218527.aspx

Very interesting blog, indeed.

Best regards,
Jean-Marc

> Another point you made is dead on, and I'd like to highlight it because it's the
> source of many "why can't I get access to all of my memory" questions...
>
> > some applications (or at least specific functions of these applications)
> > may require continuous block of memory
>
> Mathematica requires contiguous blocks of memory for very large packed arrays
> (that the memory is contiguous is part of what makes packed arrays so efficient,
> so this really is a feature).  The memory usage of Mathematica begins fragmented
> as a result of where certain system DLLs decide to load themselves in the
> virtual address range.  And, to be honest, there's a bit of fragmentation that's
> caused by our own loading of code in less than optimal locations (specific to
> Windows...we're looking into this for a future release).  But fragmentation is
> often going to be induced by the history of memory usage during the session.
>
> If you anticipate using large blocks of memory on a 32-bit system, it's best to
> allocate them as early in the session as possible to avoid this fragmentation.
> But it really is better to get a 64-bit system.  Such systems extremely
> affordable these days (although some vendors make it very hard or confusing to
> get a 64-bit version of Windows).  64-bit Mathematica runs very well.  Many
> people inside Wolfram, myself included, run it daily.
>
> And, incidentally, since this thread started off dealing with large files,
> rather than large memory allocations...yes, it ought to be the case (but isn't)
> that 32-bit versions should be able to deal with very large files.  Even though
> you couldn't expect to load files in their entirety, it's certainly reasonable
> to want to call SetStreamPosition and load pieces of them.  It's a limitation of
> Mathematica, and one that affects both the front end and kernel, although it
> really isn't a very easy problem to fix.  Many 32-bit applications suffer from
> similar issues, and it requires a lot of very intentional and difficult
> (particularly for large applications) engineering to overcome it.
>
> I've often seen "support of very large files" (meaning, of course, >2G) touted
> as a major feature (rather than a minor bug fix) on the short list of features
> for new versions of various industrial applications.
>
> Sincerely,
>
> John Fultz
> jfultz at wolfram.com
> User Interface Group
> Wolfram Research, Inc.
>
>
>
> On Thu, 15 Nov 2007 05:30:07 -0500 (EST), Jean-Marc Gulliet wrote:
> > [My reply is at the bottom of this email.]
> >
> > rych wrote:
> >
> >> Thanks for response
> >> Mathematica 6.0.1.0
> >> Windows XP Pro SP2 32 bit
> >> 2GB of RAM, Commit charge: 2113/5977M.
> >> Perhaps, I should increase the page file, say to 2x3GB? It's if the
> >> FileByteCount actually loads the whole file into memory to find out
> >> about its size. I'll report later whether it worked. Ultimately, I'd
> >> like to know the size limit and how to load such big files by parts.
> >> Igor
> >>
> >> On 10 Nov, 08:51, Jean-Marc Gulliet <jeanmarc.gull... at gmail.com>
> >> wrote:
> >>> rych wrote:
> >>>> Is there a maximum file size permitted in Mathematica? Because it
> >>>> seems I can't work with my 3GB file,
> >>> <snip>
> >>>
> >>> It all depends on what version of Mathematica you are using (32 or 64
> >>> bits), your operating system, the available memory you have when
> >>> importing the file, possibly the type of data imported...
> >>>
> >>> Thus the following questions: What version of Mathematica do you use?
> >>> What platform (hardware and operating system do you use? How much
> >>> memory
> >>> is available for Mathematica (or are you running many applications
> >>> simultaneously or just Mathematica)?
> >>>
> > Summary: if Mathematica, or any other applications, must load 3GB of
> > data into memory, this is doomed to failed on a 32-bit system.
> >
> > The address space of any 32-bit system, regardless of its operating
> > system (Windows, Linux, MacOs, Solaris, UNIX, etc.) or chip maker
> > (Intel, AMD, PowerPC, etc.), is limited to 4GB (i.e. the smallest
> > address is 0 and the largest is (2^32)-1 = 4,294,967,295). This limit is
> > due to the size of the address bus (32-bit width), bus that allows the
> > exchange of data between the processor and the main memory.
> >
> > On Windows, the main memory is managed as virtual memory, i.e. the main
> > memory is made of physical ram and a page file. Note that the
> > addressable virtual memory cannot exceeds 4GB even though the page file
> > may be created as large as you wish.
> >
> > In addition of this limitation to a maximum of 4GB of addressable
> > memory, the address space is variously divided between the operating
> > system and the user(s). Still on Windows XP, the kernel (system memory)
> > gets 2GB and the applications get the other 2GB. (It is possible on
> > certain circumstances to increase this ratio in favor of the user
> > applications up to 3GB or so.)
> >
> > Also, applications running concurrently must share this 2GB space and
> > some applications (or at least specific functions of these applications)
> > may require continuous block of memory (similarly to a hard disk, the
> > main memory will become fragmented during a work session).
> >
> > Conclusion: if Mathematica (or any other applications) must load 3GB of
> > data into memory, this is doomed to failed on a 32-bit system.
> >
> > The following pages may be particularly worth reading:
> >
> > "Windows XP Professional x64 Edition, From Wikipedia, the free
> > encyclopedia"
> >
> > http://en.wikipedia.org/wiki/Windows_XP_Professional_x64_Edition
> >
> > particularly the first section is of uttermost importance regarding
> > memory differences between 32- and 64-bit architectures.
> >
> > "RAM, Virtual Memory, Pagefile and all that stuff"
> > http://support.microsoft.com/default.aspx/kb/555223
> >
> > Regards,


  • Prev by Date: Re: Message: "Numerical interation converging too slowly"
  • Next by Date: Re: RegionPlot ignores the AxesLabel Option?
  • Previous by thread: Re: negative FileByteCount
  • Next by thread: Re: negative FileByteCount