Parallelize & Functions That Remember Values They Have Found

*To*: mathgroup at smc.vnet.net*Subject*: [mg115490] Parallelize & Functions That Remember Values They Have Found*From*: Guido Walter Pettinari <coccoinomane at gmail.com>*Date*: Wed, 12 Jan 2011 04:08:46 -0500 (EST)

Dear group, I am starting to discover the magic behind Parallelize and ParallelTable, but I still have got many problems. The latest one occurred when I tried to parallelize a function that is supposed to store his values, i.e. those defined as f[x_] := f[x] = ..... You can reproduce my problem by running the following snippet twice: f[n_] := f[n] = Prime[n] DistributeDefinitions[f]; result = ParallelTable[f[n], {n, 500000}] // AbsoluteTiming; elapsed = result[[1]] On my machine, the first execution takes 2 seconds. Since I defined f as f[x_]:=f[x], I expect the second execution to take much less than that, but it actually takes around 1.8s. The third one takes something less than that (say 1.4s), and so on. After many executions, the execution time stabilizes to 0.6 seconds. Incidentally, 0.6 seconds is the time that a normal Table takes (on the second execution) to run the same code: Exit[] f[n_] := f[n] = Prime[n] result = Table[f[n], {n, 500000}] // AbsoluteTiming; elapsed = result[[1]] It looks like my 4 kernels are storing the downvalues of f[x] separately, so that each of them stores only a (random) quarter of the f-values every time the code is run. When all of them have all of the 500.000 f-values, which happens after many executions, the execution time finally reaches 0.6s. Is there a way to make all the f-values stored by the 4 kernels available? Maybe a function that "collapses" all the information gathered by the kernels into the main kernel, i.e. a DeDistributeDefinitions function? Or maybe a way to access the memory of all 4 kernels? I tried to SetSharedFunction on f[x], but it just made the calculation extremely long. I will be grateful for any suggestion. Thank you for your attention, Guido W. Pettinari