Re: Database memory usage
- To: mathgroup at smc.vnet.net
- Subject: [mg109437] Re: Database memory usage
- From: David Reiss <dbreiss at gmail.com>
- Date: Tue, 27 Apr 2010 04:06:11 -0400 (EDT)
- References: <hr3tj2$hdo$1@smc.vnet.net>
Hi Rui, There are two parts to the answer to your question. The first one has to do directly with the Java issue. DatabaseLink uses Java as an intermediary to communicate with the database. Because of this any memory restrictions on the Java process will limit the amount of data that you can move between Mathematica and the database at any one time. One way around this that is scalable is to bring the data into or send the data to the database in chunks so that the Java heap size is not exceeded. A simple approach to this is to give your database table an auto-indexing column and to use the values in that column to select the portions of the data sequentially (when getting data from the database). And of course when sending data to the database only send it in smaller sets of data. The general rule though is that if you try to send a vary large amount of data to a database all at one time (or retrieve a very large amount) you will run in to problems eventually with the Java memory allocation. Another possible issue with hsqldb that you may run into is that (I believe) hsqldb is an entirely memory resident database: in contrast to mySQL and others. So you may run into issues with that baed on the size of the database tables and the size of your memory. This is independent of the java issue. A non-scalable way to address the Java issue can be to reset the default heap size that Java uses on your computer (this is independent of amy Mathematica settings). Here is a way to do this, setting the Jave heap size to 1 Gigabyte. But you must do this befor eloading DatabaseLink: Needs["JLink`"]; InstallJava[]; ReinstallJava[CommandLine -> "java", JVMArguments -> "-Xmx1g"]; LoadJavaClass["java.lang.Runtime"]; Print["The maximum memory for Java is currently "<>ToString[java`lang`Runtime`getRuntime[][maxMemory[]]]] And here is a link that tells you a bit about how to specify the java heap settings: http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html But remember that this only resets the Java heap size.... if you try to pull too much data from the database via DatabaseLink, you will ultimately exceed it.... Hope this helps, David http://scientificarts.com/worklife On Apr 26, 7:31 am, Rui <rui.r... at gmail.com> wrote: > I'm new to databases. > I want to put a big list in a database, to later work on it. > It all worked fine, using the default hsqldb that comes in > Mathematica, but I couldn't insert more than few thousand elements at > a time, and then Link problems or java heap problems. > Finally, I couldn't even open the connection. Java runs out of memory. > Memory usage of javaw process is less than 300MB when it runs out. > In any case, I thought that the databases were, or could be, stored in > the disk, so they can really be big and be a real big advantage over > the list. How can that be done? > If the question is too newbie and you feel I need to read a tutorial > on something, hehe, feel free to send me a link to a good one, hehe > > Rui Rojo