MathGroup Archive 2010

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Database memory usage

  • To: mathgroup at smc.vnet.net
  • Subject: [mg109437] Re: Database memory usage
  • From: David Reiss <dbreiss at gmail.com>
  • Date: Tue, 27 Apr 2010 04:06:11 -0400 (EDT)
  • References: <hr3tj2$hdo$1@smc.vnet.net>

Hi Rui,

There are two parts to the answer to your question.  The first one has
to do directly with the Java issue.  DatabaseLink uses Java as an
intermediary to communicate with the database.  Because of this any
memory restrictions on the Java process will limit the amount of data
that you can move between Mathematica and the database at any one
time.  One way around this that is scalable is to bring the data into
or send the data to the database in chunks so that the Java heap size
is not exceeded.   A simple approach to this is to give your database
table an auto-indexing column and to use the values in that column to
select the portions of the data sequentially (when getting data from
the database).  And of course when sending data to the database only
send it in smaller sets of data.  The general rule though is that if
you try to send a vary large amount of data to a database all at one
time (or retrieve a very large amount) you will run in to problems
eventually with the Java memory allocation.

Another possible issue with hsqldb that you may run into is that (I
believe) hsqldb is an entirely memory resident database: in contrast
to mySQL and others.  So you may run into issues with that baed on the
size of the database tables and the size of your memory.  This is
independent of the java issue.

A non-scalable way to address the Java issue can be to reset the
default heap size that Java uses on your computer (this is independent
of amy Mathematica settings).    Here is a way to do this, setting the
Jave heap size to 1 Gigabyte.  But you must do this befor eloading
DatabaseLink:

Needs["JLink`"];

InstallJava[];

ReinstallJava[CommandLine -> "java", JVMArguments -> "-Xmx1g"];

LoadJavaClass["java.lang.Runtime"];

Print["The maximum memory for Java is currently
"<>ToString[java`lang`Runtime`getRuntime[][maxMemory[]]]]


And here is  a link that tells you a bit about how to specify the java
heap settings:

http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html

But remember that this only resets the Java heap size....   if you try
to pull too much data from the database via  DatabaseLink, you will
ultimately exceed it....

Hope this helps,

David
http://scientificarts.com/worklife

On Apr 26, 7:31 am, Rui <rui.r... at gmail.com> wrote:
> I'm new to databases.
> I want to put a big list in a database, to later work on it.
> It all worked fine, using the default hsqldb that comes in
> Mathematica, but I couldn't insert more than few thousand elements at
> a time, and then Link problems or java heap problems.
> Finally, I couldn't even open the connection. Java runs out of memory.
> Memory usage of javaw process is less than 300MB when it runs out.
> In any case, I thought that the databases were, or could be, stored in
> the disk, so they can really be big and be a real big advantage over
> the list. How can that be done?
> If the question is too newbie and you feel I need to read a tutorial
> on something, hehe, feel free to send me a link to a good one, hehe
>
> Rui Rojo



  • Prev by Date: Re: Context Problem
  • Next by Date: Re: Context Problem
  • Previous by thread: Re: Database memory usage
  • Next by thread: Re: Database memory usage