[Date Index] [Thread Index] [Author Index]

Re: negative FileByteCount

• To: mathgroup at smc.vnet.net
• Subject: [mg83201] Re: negative FileByteCount
• From: Jean-Marc Gulliet <jeanmarc.gulliet at gmail.com>
• Date: Thu, 15 Nov 2007 05:30:07 -0500 (EST)
• Organization: The Open University, Milton Keynes, UK
• References: <fh1c35$boh$1@smc.vnet.net><fh3rfa$115$1@smc.vnet.net> <fheg8a$lvg$1@smc.vnet.net>

[My reply is at the bottom of this email.]

rych wrote:

> Thanks for response
> Mathematica 6.0.1.0
> Windows XP Pro SP2 32 bit
> 2GB of RAM, Commit charge: 2113/5977M.
> Perhaps, I should increase the page file, say to 2x3GB? It's if the
> FileByteCount actually loads the whole file into memory to find out
> about its size. I'll report later whether it worked. Ultimately, I'd
> like to know the size limit and how to load such big files by parts.
> Igor
> 
> On 10 Nov, 08:51, Jean-Marc Gulliet <jeanmarc.gull... at gmail.com>
> wrote:
>> rych wrote:
>>> Is there a maximum file size permitted in Mathematica? Because it
>>> seems I can't work with my 3GB file,
>> <snip>
>>
>> It all depends on what version of Mathematica you are using (32 or 64
>> bits), your operating system, the available memory you have when
>> importing the file, possibly the type of data imported...
>>
>> Thus the following questions: What version of Mathematica do you use?
>> What platform (hardware and operating system do you use? How much memory
>> is available for Mathematica (or are you running many applications
>> simultaneously or just Mathematica)?

Summary: if Mathematica, or any other applications, must load 3GB of 
data into memory, this is doomed to failed on a 32-bit system.

The address space of any 32-bit system, regardless of its operating 
system (Windows, Linux, MacOs, Solaris, UNIX, etc.) or chip maker 
(Intel, AMD, PowerPC, etc.), is limited to 4GB (i.e. the smallest 
address is 0 and the largest is (2^32)-1 = 4,294,967,295). This limit is 
due to the size of the address bus (32-bit width), bus that allows the 
exchange of data between the processor and the main memory.

On Windows, the main memory is managed as virtual memory, i.e. the main 
memory is made of physical ram and a page file. Note that the 
addressable virtual memory cannot exceeds 4GB even though the page file 
may be created as large as you wish.

In addition of this limitation to a maximum of 4GB of addressable 
memory, the address space is variously divided between the operating 
system and the user(s). Still on Windows XP, the kernel (system memory) 
gets 2GB and the applications get the other 2GB. (It is possible on 
certain circumstances to increase this ratio in favor of the user 
applications up to 3GB or so.)

Also, applications running concurrently must share this 2GB space and 
some applications (or at least specific functions of these applications) 
may require continuous block of memory (similarly to a hard disk, the 
main memory will become fragmented during a work session).

Conclusion: if Mathematica (or any other applications) must load 3GB of 
data into memory, this is doomed to failed on a 32-bit system.

The following pages may be particularly worth reading:

"Windows XP Professional x64 Edition, From Wikipedia, the free encyclopedia"

http://en.wikipedia.org/wiki/Windows_XP_Professional_x64_Edition

particularly the first section is of uttermost importance regarding 
memory differences between 32- and 64-bit architectures.

"RAM, Virtual Memory, Pagefile and all that stuff"
http://support.microsoft.com/default.aspx/kb/555223

Regards,
-- 
Jean-Marc

• Follow-Ups:
  • Re: Re: negative FileByteCount
    • From: "Jean-Marc Gulliet" <jeanmarc.gulliet@gmail.com>