Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2010

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Database memory usage

  • To: mathgroup at smc.vnet.net
  • Subject: [mg109438] Re: Database memory usage
  • From: Albert Retey <awnl at gmx-topmail.de>
  • Date: Tue, 27 Apr 2010 04:06:22 -0400 (EDT)
  • References: <hr3tj2$hdo$1@smc.vnet.net>

Hi,

> I'm new to databases.
> I want to put a big list in a database, to later work on it.
> It all worked fine, using the default hsqldb that comes in
> Mathematica, but I couldn't insert more than few thousand elements at
> a time, and then Link problems or java heap problems.
> Finally, I couldn't even open the connection. Java runs out of memory.
> Memory usage of javaw process is less than 300MB when it runs out.

the java vm runs with a fixed size of memory that it is allowed to use,
I think that something between 200 and 300 MB is the default so that is
what you see it runs out of. This setting can be adjusted, and I think
there has been an explanation of how to do it in a post some time ago.

> In any case, I thought that the databases were, or could be, stored in
> the disk, so they can really be big and be a real big advantage over
> the list. How can that be done?

As far as I remember the hsql default is to store tables in memory if
possible and writes just a log to disk from which it can regenerate the
data in memory. There is a special way to define tables to always be
written to disk. I don't exactly remember the details and don't know
whether that could help with your problem but you should find details in
the documentation for hsql.

I think if you try to store the huge list in one go (and read it back
that way) you would be better of to use a binary format to store the
list in one file. You could try the Dump (.mx) or Wolfram Data Exchange
(wdx) formats, which are both documented. In my opinion using a database
would only make sense if your data has some kind of structure and you
would want to work only on a subset of the huge list at a time that
would be easy to be extracted from the total data with a SQLSelect.

> If the question is too newbie and you feel I need to read a tutorial
> on something, hehe, feel free to send me a link to a good one, hehe

Maybe you want to explain some more details about your huge list, what
you want to do with it and why you need to store it on disk...

hth,

albert


  • Prev by Date: Re: Database memory usage
  • Next by Date: Re: Context Problem
  • Previous by thread: Re: Database memory usage
  • Next by thread: Re: Database memory usage