Re: Parallelize & Functions That Remember Values They Have Found
- To: mathgroup at smc.vnet.net
- Subject: [mg115551] Re: Parallelize & Functions That Remember Values They Have Found
- From: Guido Walter Pettinari <coccoinomane at gmail.com>
- Date: Fri, 14 Jan 2011 06:18:56 -0500 (EST)
- References: <igmd11$d1$1@smc.vnet.net>
Thank you very much for both answers!
I took inspiration from your example, Thomas, and I found a solution
to my original issue. I just need to add to the original code I
posted (that is
f[n_] := f[n] = Prime[n]
DistributeDefinitions[f];
result = ParallelTable[f[n], {n, 500000}] // AbsoluteTiming;
elapsed = result[[1]]
) the following lines:
DownValues[f] = Flatten[ParallelEvaluate[DownValues[f]]];
SetSharedFunction[f]
The first line collects the data associated with the function f from
the parallel kernel into the main one, so that whenever I call f[n]
from the main kernel, it does not need to be recomputed.
The second line sets f as a shared function, thus allowing the
parallel kernels to access all the data I just stored into the main
kernel. In this way I do not need to use up memory on the parallel
kernels, since everything is stored in the main kernel. The drawback
is that when the parallel kernels are asked to re-compute f[n] (for
example re-executing the first 4 lines of code), data needs to be
transfered from the main kernel.
This is a major issue in our example, since transferring half-million
numbers from the main to the parallel kernels takes longer than re-
computing everything. However, it's not an issue for me, since I need
to store interpolation tables (i.e. solutions from NDSolve), which are
usually very short.
As a side note, if one is interested only in accessing the data in the
main kernel, the SetSharedFunction[f] command can be avoided.
Please let me know if you find a better solution, and thank you again
for the replies.
Ciao,
Guido
On Jan 13, 8:27 am, thomas <thomas.mue... at gmail.com> wrote:
> Dear Guido,
>
> I have faced a similar problem recently. As a way around this, I collecte=
d the definitions known to the remote kernels in the following way:
>
> f[n_] := f[n] = Prime[n]
> DistributeDefinitions[f];
> ParallelEvaluate[f[n], {n, 500000}];(*now all f's are known remotely*)
> DownValues[f]=Flatten[ParallelEvaluate[DownValues[f]]];(*now all f's ar=
e known centrally*)
> result = Table[f[n], {n, 500000}];
>
> This collection of data can take quite some time and eat up the advantage=
s you gain by parallelization. So it is only worth doing this if your real =
code gains enough speed by parallel evaluation. The best is to experiment w=
ith that!
>
> Even though it works, it seems quite cumbersome to me. I feel that there =
should be a better way.
>
> thomas
>
>
>
>
>
>
>
> On Wednesday, January 12, 2011 10:08:44 AM UTC+1, Guido Walter Pettinari =
wrote:
> > Dear group,
>
> > I am starting to discover the magic behind Parallelize and
> > ParallelTable, but I still have got many problems. The latest one
> > occurred when I tried to parallelize a function that is supposed to
> > store his values, i.e. those defined as f[x_] := f[x] = .....
>
> > You can reproduce my problem by running the following snippet twice:
>
> > f[n_] := f[n] = Prime[n]
> > DistributeDefinitions[f];
> > result = ParallelTable[f[n], {n, 500000}] // AbsoluteTiming;
> > elapsed = result[[1]]
>
> > On my machine, the first execution takes 2 seconds. Since I defined =
f
> > as f[x_]:=f[x], I expect the second execution to take much less than
> > that, but it actually takes around 1.8s. The third one takes
> > something less than that (say 1.4s), and so on. After many
> > executions, the execution time stabilizes to 0.6 seconds.
>
> > Incidentally, 0.6 seconds is the time that a normal Table takes (on
> > the second execution) to run the same code:
>
> > Exit[]
> > f[n_] := f[n] = Prime[n]
> > result = Table[f[n], {n, 500000}] // AbsoluteTiming;
> > elapsed = result[[1]]
>
> > It looks like my 4 kernels are storing the downvalues of f[x]
> > separately, so that each of them stores only a (random) quarter of the
> > f-values every time the code is run. When all of them have all of th=
e
> > 500.000 f-values, which happens after many executions, the execution
> > time finally reaches 0.6s.
>
> > Is there a way to make all the f-values stored by the 4 kernels
> > available? Maybe a function that "collapses" all the information
> > gathered by the kernels into the main kernel, i.e. a
> > DeDistributeDefinitions function? Or maybe a way to access the memor=
y
> > of all 4 kernels? I tried to SetSharedFunction on f[x], but it just
> > made the calculation extremely long.
>
> > I will be grateful for any suggestion.
>
> > Thank you for your attention,
>
> > Guido W. Pettinari