Distributing "Large" Expressions to Remote Nodes
- To: mathgroup at smc.vnet.net
- Subject: [mg107856] Distributing "Large" Expressions to Remote Nodes
- From: Eric Brown <eric.c.brown at mac.com>
- Date: Mon, 1 Mar 2010 04:42:38 -0500 (EST)
Dear MathGroup,
I would like to send "large" expressions between a master and a parallel slave on a remote machine:
slaves = LaunchKernels[]
(* {KernelObject[1, "brown.chem.luc.edu"]} *)
Do[
data=Table[RandomReal[],{l}];
DistributeDefinitions[data];
returnValue=ParallelEvaluate[Length[data]];
Print[{l,ByteCount[data], returnValue}];
,{l,10^5,10^7,10^5}]
On my Mac OS X machines (10.5 and 10.6) I can only get up to about a 28 megabyte message before transfer stops. It is a similar situation between two Ubuntu machines.
My machines have at least 16 gig of RAM, and they should have no problems with this size of messages. I have played with ReinstallJava settings (trying to enlarge the heap) but I have still had no luck.
I contacted Wolfram about this some time ago, and I seem to recall that the suggestion was to use a file-based approach. I have tried this, but I run into a similar situation. I can only load only relatively small files on remote nodes, including from .wdx, .m, .h5, etc.
Would someone mind trying the above code on their systems? My eventual goal is to run code across very large Beowulf type clusters. If you have a solution to this problem, then I would be most grateful.
Best regards,
Eric