[Date Index] [Thread Index] [Author Index]

Re: General question on handling very large datasets in Mathematica

• To: mathgroup at smc.vnet.net
• Subject: [mg48844] Re: General question on handling very large datasets in Mathematica
• From: Jens-Peer Kuska <kuska at informatik.uni-leipzig.de>
• Date: Sat, 19 Jun 2004 04:30:44 -0400 (EDT)
• Organization: Universitaet Leipzig
• References: <cau1o5$akh$1@smc.vnet.net>
• Reply-to: kuska at informatik.uni-leipzig.de
• Sender: owner-wri-mathgroup at wolfram.com

Hi,

I have only large images with up to 512 MByte data.
You should use the 64 versions of Mathematica because
a 64 operating system is not limited to 4 GByte memory.
and you should try to avoid to copy the data to often
when you call your functions (Hold attribute)

Regards
  Jens

Mark Coleman wrote:
> 
> Greetings,
> 
> I am about to begin some research work with a very large dataset --
> several million records (though each record is relatively simple). I
> plan to use Mathematica to estimate a number of statistical models using
> various subsets of this data. I was wondering if other MathGroup have
> any experiences they could share using Mathematica to handle large datasets. My
> particular concern is Import-Export and general reading of large data
> sets in Mathematica. I can have this data in a variety of formats, from pure
> ASCII to any number of RDBS formats.
> 
> I apologize for the vagueness of this posting.
> 
> Regards,
> 
> Mark