parallelization - Efficiency of ParallelDo when a global variable./matrix is being updated

(Disclaimer: I realize that the addition of matrices a and b below can be done very fast just by typing a+b. The code below is just meant to illustrate behavior similar to what I observe in much more complicated code, too lengthy to reproduce here).

Suppose I want to add matrices a and b

n = 50;

I can do it via a double loop in a single kernel, this takes 0.02 seconds:

a = RandomReal[{}, {n, n}];
b = RandomReal[{}, {n, n}];
c1 = RandomReal[{}, {n, n}];
Do[
  Do[

   c1[[i, j]] = b[[i, j]] + a[[i, j]],
   {j, 1, n}],
 {i, 1, n}] // AbsoluteTiming

Out[363]= {0.0240014, Null}

I can try to parallelize the outer loop -- I use 4 kernels -- this takes 25 seconds -- ouch -- presumably due to the fact that the master kernel is in constant communication with the various kernels, updating c2.

SetSharedVariable[c2];
c2 = RandomReal[{}, {n, n}];
ParallelDo[

  Do[
   c2[[i, j]] = b[[i, j]] + a[[i, j]],
   {j, 1, n}],
  {i, 1, n}] // AbsoluteTiming

Out[366]= {25.8984813, Null}

I can try to reduce communication to an absolute minimum, by introducing a temporary array on each kernel. Surprisingly, this still requires 0.3 seconds.

SetSharedVariable[c3];
UnsetShared[vtemp];

c3 = RandomReal[{}, {n, n}];
ParallelDo[
  vtemp = Table[0., {n}];
  Do[
   vtemp[[j]] = b[[i, j]] + a[[i, j]],
   {j, 1, n}];
  c3[[i]] = vtemp,
  {i, 1, n}, Method -> "CoarsestGrained"] // AbsoluteTiming

Out[370]= {0.2940168, Null}

Again, my own code is far more complicated. But it shares with this example the need to distribute the task of updating a piece of a matrix. Updating the global variable/matrix invariably takes more time than executing the code on a single processor, even when the calculations on each processor are far more complex that in the above. Are there techniques to mitigate this problem?

Answer

Is there any reason you can't directly construct the Table, as in:

DistributeDefinitions[a,b]
c1 = ParallelTable[b[[i, j]] + a[[i, j]], {j, 1, n}, {i, 1, n}]

On my machine, with n=500, Table[b[[i, j]] + a[[i, j]], {j, 1, n}, {i, 1, n}];// AbsoluteTiming takes 0.59 s, while ParallelTable takes only 0.16 s. With n=1000, Table takes 2.1s while ParallelTable takes only 0.47s.

For future reference, unlike in MATLAB or other languages, in Mathematica you do not need to first "initialize" the table, and then fill it.