Re: General question on handling very large datasets in Mathematica
- To: mathgroup at smc.vnet.net
- Subject: [mg48844] Re: General question on handling very large datasets in Mathematica
- From: Jens-Peer Kuska <kuska at informatik.uni-leipzig.de>
- Date: Sat, 19 Jun 2004 04:30:44 -0400 (EDT)
- Organization: Universitaet Leipzig
- References: <cau1o5$akh$1@smc.vnet.net>
- Reply-to: kuska at informatik.uni-leipzig.de
- Sender: owner-wri-mathgroup at wolfram.com
Hi, I have only large images with up to 512 MByte data. You should use the 64 versions of Mathematica because a 64 operating system is not limited to 4 GByte memory. and you should try to avoid to copy the data to often when you call your functions (Hold attribute) Regards Jens Mark Coleman wrote: > > Greetings, > > I am about to begin some research work with a very large dataset -- > several million records (though each record is relatively simple). I > plan to use Mathematica to estimate a number of statistical models using > various subsets of this data. I was wondering if other MathGroup have > any experiences they could share using Mathematica to handle large datasets. My > particular concern is Import-Export and general reading of large data > sets in Mathematica. I can have this data in a variety of formats, from pure > ASCII to any number of RDBS formats. > > I apologize for the vagueness of this posting. > > Regards, > > Mark