Re: Database memory usage
- To: mathgroup at smc.vnet.net
- Subject: [mg109427] Re: Database memory usage
- From: David Bailey <dave at removedbailey.co.uk>
- Date: Tue, 27 Apr 2010 04:04:21 -0400 (EDT)
- References: <hr3tj2$hdo$1@smc.vnet.net>
Rui wrote: > I'm new to databases. > I want to put a big list in a database, to later work on it. > It all worked fine, using the default hsqldb that comes in > Mathematica, but I couldn't insert more than few thousand elements at > a time, and then Link problems or java heap problems. > Finally, I couldn't even open the connection. Java runs out of memory. > Memory usage of javaw process is less than 300MB when it runs out. > In any case, I thought that the databases were, or could be, stored in > the disk, so they can really be big and be a real big advantage over > the list. How can that be done? > If the question is too newbie and you feel I need to read a tutorial > on something, hehe, feel free to send me a link to a good one, hehe > > Rui Rojo > If you just want to write a lot of data away in chunks and re-read it later, there is no need to use a database at all: str=OpenWrite["Myfile.dat"]; Write[str,list1] .. .. .. Write[str,listN] Close[str] You could also write the data in binary form if you prefer. However, from what you say, I imagine there is some sort of memory leak in the database handling process - you should report this as a bug to Wolfram. In the meantime, if you really need to write a database, you will need to use J/Link and possibly some actual Java code to do the job correctly. David Bailey http://www.dbaileyconsultancy.co.uk