Parallelize & Functions That Remember Values They Have Found
- To: mathgroup at smc.vnet.net
- Subject: [mg115490] Parallelize & Functions That Remember Values They Have Found
- From: Guido Walter Pettinari <coccoinomane at gmail.com>
- Date: Wed, 12 Jan 2011 04:08:46 -0500 (EST)
Dear group,
I am starting to discover the magic behind Parallelize and
ParallelTable, but I still have got many problems. The latest one
occurred when I tried to parallelize a function that is supposed to
store his values, i.e. those defined as f[x_] := f[x] = .....
You can reproduce my problem by running the following snippet twice:
f[n_] := f[n] = Prime[n]
DistributeDefinitions[f];
result = ParallelTable[f[n], {n, 500000}] // AbsoluteTiming;
elapsed = result[[1]]
On my machine, the first execution takes 2 seconds. Since I defined f
as f[x_]:=f[x], I expect the second execution to take much less than
that, but it actually takes around 1.8s. The third one takes
something less than that (say 1.4s), and so on. After many
executions, the execution time stabilizes to 0.6 seconds.
Incidentally, 0.6 seconds is the time that a normal Table takes (on
the second execution) to run the same code:
Exit[]
f[n_] := f[n] = Prime[n]
result = Table[f[n], {n, 500000}] // AbsoluteTiming;
elapsed = result[[1]]
It looks like my 4 kernels are storing the downvalues of f[x]
separately, so that each of them stores only a (random) quarter of the
f-values every time the code is run. When all of them have all of the
500.000 f-values, which happens after many executions, the execution
time finally reaches 0.6s.
Is there a way to make all the f-values stored by the 4 kernels
available? Maybe a function that "collapses" all the information
gathered by the kernels into the main kernel, i.e. a
DeDistributeDefinitions function? Or maybe a way to access the memory
of all 4 kernels? I tried to SetSharedFunction on f[x], but it just
made the calculation extremely long.
I will be grateful for any suggestion.
Thank you for your attention,
Guido W. Pettinari