Services & ResourcesWolfram ForumsMathGroup Archive

[Date Index] [Thread Index] [Author Index]

Re: Tracking progress in ParallelDo

• To: mathgroup at smc.vnet.net
• Subject: [mg132599] Re: Tracking progress in ParallelDo
• From: Szabolcs HorvÃt <szhorvat at gmail.com>
• Date: Thu, 17 Apr 2014 05:10:15 -0400 (EDT)
• Delivered-to: l-mathgroup@mail-archive0.wolfram.com
• Delivered-to: l-mathgroup@wolfram.com
• Delivered-to: mathgroup-outx@smc.vnet.net
• Delivered-to: mathgroup-newsendx@smc.vnet.net
• References: <lig9n7$cfl$1@smc.vnet.net>

There was a very similar question asked on StackExchange.  You might be 
interested in the answers there.

http://mathematica.stackexchange.com/questions/1548/monitor-doesnt-work-with-paralleltable

Here's a slightly modified version of Leonid's solution:

SetSharedVariable[progress]

progress = 0;
Monitor[
  ParallelTable[progress++; Pause[0.5]; i, {i, 100}],
  progress
]

The caveat here is that SetSharedVariable causes 'progress' to be always 
evaluated on the main kernel.  Accessing the main kernel is expensive 
(it takes a long time).  So this only makes sense if:

  - it is reasonable to break the parallel evaluation into the smallest 
possible pieces (equivalent of Method -> "FinestGrained")

  - each single evaluation in your ParallelDo takes considerably longer 
than the overhead due to the callback to the main kernel

Otherwise doing this will kill performance.  Personally I do not 
recommend doing this unless you are experienced with parallelization in 
Mathematica and understand what factors affect performance.

Do check my link for more complex but also more efficient variations on 
the same idea (Eli Lansey's answer).

Szabolcs


On 2014-4-14, 5:28 , Brian Beckage wrote:
> I'm applying a function repeatedly using ParallelDo.  I would like to keep track of its progress.  I've tried using both my own counter variable as well as using the increments within the ParallelDo function (below). Neither provides a way of sequentially tracking how many of the iterations have been spawned (not completed--but just kicked off).
>
> How could I track that iteration 1, 2, 3, 4, 5 etc. have been started?  Is there a better way of monitoring progress of the ParalleDo?
>
> Thanks for your help!
>
> Brian
>
>
> USING MY OWN COUNTER: myCounter:
>
> parallelBootstrapFireMultiplier[nBoot_, fireMultiplierRange_, parmList1_,
>    nYears_, initLandscapeProbs_] := Module[{outIter, myCounter},
>
>    ParallelEvaluate[Clear[myCounter]];
>    myCounter = 0;
>    SetSharedVariable[myCounter];
>    Reap[
>     ParallelDo[
>      myCounter++;
>      Print["Starting parallelBootstrapMultiplier step ", myCounter, " out of ",
>        nBoot];
>      outIter =
>       iterFireMultiplier[fireMultiplierRange, parmList1, nYears,
>        initLandscapeProbs];
>      ParallelSow[outIter],
>      {nBoot}]
>     ]
>    ]
>
> RESULTS in:
>
> (kernel 2) Starting parallelBootstrapMultiplier step 2 out of 6
> (kernel 1) Starting parallelBootstrapMultiplier step 2 out of 6
> (kernel 1) Starting parallelBootstrapMultiplier step 4 out of 6
> (kernel 2) Starting parallelBootstrapMultiplier step 4 out of 6
> (kernel 2) Starting parallelBootstrapMultiplier step 5 out of 6
> (kernel 1) Starting parallelBootstrapMultiplier step 6 out of 6
>
> USING THE INCREMENT i FROM ParallelDo:
>
> parallelBootstrapFireMultiplier[nBoot_,fireMultiplierRange_,parmList1_,nYears_,initLandscapeProbs_]:= Module[{outIter},
> Reap[
> ParallelDo[
> Print["Starting parallelBootstrapMultiplier step ",i, " out of ",nBoot];
> outIter=iterFireMultiplier[fireMultiplierRange,parmList1,nYears,initLandscapeProbs]; (*This returns a list*)
> ParallelSow[outIter],
> {i,nBoot}]
> ]
> ]
>
> RESULTS in:
> Starting parallelBootstrapMultiplier step 1 out of 6
> Starting parallelBootstrapMultiplier step 4 out of 6
> Starting parallelBootstrapMultiplier step 5 out of 6
> Starting parallelBootstrapMultiplier step 2 out of 6
> Starting parallelBootstrapMultiplier step 3 out of 6
> Starting parallelBootstrapMultiplier step 6 out of 6
>