Mathematica 9 is now available
Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2011

[Date Index] [Thread Index] [Author Index]

Search the Archive

Parallelize & Functions That Remember Values They Have Found

  • To: mathgroup at smc.vnet.net
  • Subject: [mg115490] Parallelize & Functions That Remember Values They Have Found
  • From: Guido Walter Pettinari <coccoinomane at gmail.com>
  • Date: Wed, 12 Jan 2011 04:08:46 -0500 (EST)

Dear group,

I am starting to discover the magic behind Parallelize and
ParallelTable, but I still have got many problems.  The latest one
occurred when I tried to parallelize a function that is supposed to
store his values, i.e. those defined as f[x_] := f[x] = .....

You can reproduce my problem by running the following snippet twice:

f[n_] := f[n] = Prime[n]
DistributeDefinitions[f];
result = ParallelTable[f[n], {n, 500000}] // AbsoluteTiming;
elapsed = result[[1]]

On my machine, the first execution takes 2 seconds.  Since I defined f
as f[x_]:=f[x], I expect the second execution to take much less than
that, but it actually takes around 1.8s.  The third one takes
something less than that (say 1.4s), and so on.  After many
executions, the execution time stabilizes to 0.6 seconds.

Incidentally, 0.6 seconds is the time that a normal Table takes (on
the second execution) to run the same code:

Exit[]
f[n_] := f[n] = Prime[n]
result = Table[f[n], {n, 500000}] // AbsoluteTiming;
elapsed = result[[1]]

It looks like my 4 kernels are storing the downvalues of f[x]
separately, so that each of them stores only a (random) quarter of the
f-values every time the code is run.  When all of them have all of the
500.000 f-values, which happens after many executions, the execution
time finally reaches 0.6s.

Is there a way to make all the f-values stored by the 4 kernels
available?  Maybe a function that "collapses" all the information
gathered by the kernels into the main kernel, i.e. a
DeDistributeDefinitions function?  Or maybe a way to access the memory
of all 4 kernels?  I tried to SetSharedFunction on f[x], but it just
made the calculation extremely long.

I will be grateful for any suggestion.

Thank you for your attention,

Guido W. Pettinari


  • Prev by Date: Re: Simple PDE with side conditions
  • Next by Date: Re: Cubic equations again...
  • Previous by thread: Re: Rudimentary Export Question
  • Next by thread: Re: Parallelize & Functions That Remember Values They Have Found