[Date Index] [Thread Index] [Author Index]

Re: Database memory usage

• To: mathgroup at smc.vnet.net
• Subject: [mg109427] Re: Database memory usage
• From: David Bailey <dave at removedbailey.co.uk>
• Date: Tue, 27 Apr 2010 04:04:21 -0400 (EDT)
• References: <hr3tj2$hdo$1@smc.vnet.net>

Rui wrote:
> I'm new to databases.
> I want to put a big list in a database, to later work on it.
> It all worked fine, using the default hsqldb that comes in
> Mathematica, but I couldn't insert more than few thousand elements at
> a time, and then Link problems or java heap problems.
> Finally, I couldn't even open the connection. Java runs out of memory.
> Memory usage of javaw process is less than 300MB when it runs out.
> In any case, I thought that the databases were, or could be, stored in
> the disk, so they can really be big and be a real big advantage over
> the list. How can that be done?
> If the question is too newbie and you feel I need to read a tutorial
> on something, hehe, feel free to send me a link to a good one, hehe
> 
> Rui Rojo
> 
If you just want to write a lot of data away in chunks and re-read it 
later, there is no need to use a database at all:

str=OpenWrite["Myfile.dat"];


Write[str,list1]
..
..
..
Write[str,listN]
Close[str]

You could also write the data in binary form if you prefer.

However, from what you say, I imagine there is some sort of memory leak 
in the database handling process - you should report this as a bug to 
Wolfram. In the meantime, if you really need to write a database, you 
will need to use J/Link and possibly some actual Java code to do the job 
correctly.

David Bailey
http://www.dbaileyconsultancy.co.uk