RE: Re: accessing large files
- To: mathgroup at smc.vnet.net
- Subject: [mg15886] RE: [mg15842] Re: [mg15707] accessing large files
- From: "Jean-Marie THOMAS" <jmthomas at agat.net>
- Date: Wed, 17 Feb 1999 23:33:36 -0500
- Sender: owner-wri-mathgroup at wolfram.com
I performed some tests on huge files (two dimensional array of random numbers), and this is a short summury: On a Win95 system with 32M ram, handling a 6M file is a maximum, since using Put or DumpSave takes an incredibal long time. On a Winnt4 with 128M ram, a 10M file is okay, but saving times tend to increase. So it mostly depends on the ram you have. If your ram is not enough, your operations might never end! Hope this helps, **************************************** Jean-Marie THOMAS mailto:jmthomas at agat.net Conseil et Audit en Ingenierie de Calcul Strasbourg, France http://www.agat.net **************************************** -----Original Message----- From: Tomas Garza [mailto:tgarza at mail.internet.com.mx] To: mathgroup at smc.vnet.net Subject: [mg15886] [mg15842] Re: [mg15707] accessing large files Chuck McCallum wrote: >Is there a way to read in parts of very large files (in my case about >130 megs) using mathematica? I'm trying SetStreamPosition, but it gives >me an error when I go to read in a byte, because I'm reading past the >end of the buffer. Chuck, My naive advice is that you use an appropriate database manager -- something like Microsoft Access --, which can easily handle files that size (I have used it without any hitch with precisely that size files) in order to select and prepare those parts of your files which you want to be read on any given session, and save them as text with tab delimiters, which you can then read into Mathematica with ReadList. This is straightforward and painless. Good luck, Tomas Garza Mexico City