General question on handling very large datasets in Mathematica
- To: mathgroup at smc.vnet.net
- Subject: [mg48827] General question on handling very large datasets in Mathematica
- From: Mark Coleman <mark at markscoleman.com>
- Date: Fri, 18 Jun 2004 02:12:49 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
Greetings, I am about to begin some research work with a very large dataset -- several million records (though each record is relatively simple). I plan to use Mathematica to estimate a number of statistical models using various subsets of this data. I was wondering if other MathGroup have any experiences they could share using Mathematica to handle large datasets. My particular concern is Import-Export and general reading of large data sets in Mathematica. I can have this data in a variety of formats, from pure ASCII to any number of RDBS formats. I apologize for the vagueness of this posting. Regards, Mark