Skip to main content

parallelization - ParallelMap performs badly inside function



I'm trying to parallelize a very simple code that performs a stochastic gradient descent optimization (on a dual core cpu w. hyperthreading, so 4 parallel kernels).


I use ParallelMap to compute the gradient over a randomly sampled subset of variables. I am puzzled by the fact that when the EXACT SAME definition is used inside or outside a function the performance sees a dramatic drop.


These are the definitions, where SGDgrad and SGDgradPar only differ for the use of Map vs ParallelMap


MyDer[strenghts_, efforts_, i_] := Module[{sum},
sum = strenghts.efforts;
2*strenghts[[i]]^2*efforts[[i]]/sum -
strenghts[[i]] ((strenghts^2).(efforts^2))/sum^2
]
SGDgrad[strenghts_, efforts_, rsamp_] :=
SparseArray[

Map[# -> MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp], {Length@strenghts,
Length@strenghts[[1]]}]
SGDgradPar[strenghts_, efforts_, rsamp_] :=
SparseArray[
ParallelMap[# ->
MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp,
Method -> "CoarsestGrained"], {Length@strenghts,
Length@strenghts[[1]]}];


As a test this can be run on (small - 50 samples) random data:


strenghts = RandomReal[1, {20, 10000}];
efforts = RandomReal[1, {20, 10000}];

rsamp = RandomSample[Flatten[Table[{i, j}, {i, Length@strenghts}, {j,
Length@strenghts[[1]]}], 1], 50];

I try to run the same code as a function (SGDgrad and SGDgradPar) and as a simple line of code. For the parallelized version there is a 20x performance difference!


SGDgrad[strenghts, efforts, rsamp]; // AbsoluteTiming

SGDgradPar[strenghts, efforts, rsamp]; // AbsoluteTiming

{0.001407, Null}
{0.547823, Null}

SparseArray[
Map[# ->
MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp], {Length@strenghts,
Length@strenghts[[1]]}]; // AbsoluteTiming

SparseArray[
ParallelMap[# ->
MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp,
Method -> "CoarsestGrained"], {Length@strenghts,
Length@strenghts[[1]]}]; // AbsoluteTiming

{0.001153, Null}
{0.026422, Null}


If I make the dataset bigger (50000 samples):


rsamp = RandomSample[Flatten[Table[{i, j}, {i, Length@strenghts}, {j, 
Length@strenghts[[1]]}], 1], 50000];

Then timings for the same operations are:


{1.20238, Null}
{1.87731, Null}


{1.20966, Null}

{1.01134, Null}

The advantage of parallelization is very small (which I find strange) and only when batches are large. Such advantage is completely spoiled if I parallelize inside a function rather than outside...


Can someone explain these differences? Am I doing something wrong?


I have already tried distributing the definitions of the variables and MyDer, and nothing changes.



Answer



Here is a much smaller example demonstrating the difference:


In[17]:= ParallelMap[# -> 
MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp,

Method -> "CoarsestGrained"]; // AbsoluteTiming

Out[17]= {0.065301, Null}

In[13]:= With[{strenghts = strenghts, efforts = efforts},
ParallelMap[# ->
MyDer[strenghts[[All, #[[2]]]],
efforts[[All, #[[2]]]], #[[1]]] &, rsamp,
Method -> "CoarsestGrained"]
]; // AbsoluteTiming


Out[13]= {0.488711, Null}

When you use a function (or With above), strenghts and efforts are inlined into the pure function that you are mapping. These arrays are sent to the subkernels once for each evaluation. The literal value of the arrays is part of the expression describing the pure function. The pure function becomes a huge expression that is slow to transfer.


When you use variables, they are sent (distributed, see DistributeDefinitions) once and re-used multiple times. The function that you are mapping does not contain the literal value of these arrays. It only contains references to them. The function will be sent to subkernels once for each evaluation. But this time the function is a small expression that does not take a long time to transfer.





The advantage of parallelization is very small (which I find strange) and only when batches are large. Such advantage is completely spoiled if I parallelize inside a function rather than outside...



In Mathematica, parallelization involves explicitly transferring data between the main kernel and the subkernels. This transfer is expensive (i.e. it takes a long time). Parallelization is only worth it if the computation takes considerably longer than the data transfer.



Comments

Popular posts from this blog

plotting - How to draw lines between specified dots on ListPlot?

I would like to create a plot where I have unconnected dots and some connected. So far, I have figured out how to draw the dots. My code is the following: ListPlot[{{1, 1}, {2, 2}, {3, 3}, {4, 4}, {1, 4}, {2, 5}, {3, 6}, {4, 7}, {1, 7}, {2, 8}, {3, 9}, {4, 10}, {1, 10}, {2, 11}, {3, 12}, {4,13}, {2.5, 7}}, Ticks -> {{1, 2, 3, 4}, None}, AxesStyle -> Thin, TicksStyle -> Directive[Black, Bold, 12], Mesh -> Full] I have thought using ListLinePlot command, but I don't know how to specify to the command to draw only selected lines between the dots. Do have any suggestions/hints on how to do that? Thank you. Answer One possibility would be to use Epilog with Line : ListPlot[ {{1, 1}, {2, 2}, {3, 3}, {4, 4}, {1, 4}, {2, 5}, {3, 6}, {4, 7}, {1, 7}, {2, 8}, {3, 9}, {4, 10}, {1, 10}, {2, 11}, {3, 12}, {4, 13}, {2.5, 7}}, Ticks -> {{1, 2, 3, 4}, None}, AxesStyle -> Thin, TicksStyle -> Directive[Black, Bold, 12], Mesh -> Full, Epilog -> { Line[ ...

list manipulation - Selecting multiple columns from a matrix?

Sample data: data = { {{2013, 1, 1}, 24.13, 167.67, 231.82}, {{2013, 1, 2}, 32.15, 170.92, 225.99}, {{2013, 1, 3}, 35.43, 172.68, 221.67}, {{2013, 1, 4}, 36.73, 173.05, 218.32}, {{2013, 1, 5}, 58.19, 165.96, 197.05}, {{2013, 1, 6}, 69.99, 163.50, 187.52}, {{2013, 1, 7}, 71.37, 154.21, 175.58}, {{2013, 1, 8}, 72.51, 149.66, 163.25}}; I want a DateListPlot with three graphs, so for a matrix formed by columns 1 and 2, one for columns 1 and 3, and 1 for columns 1 and 4. At the moment I'm using this code: data2 = Transpose[{data[[All, 1]], data[[All, 2]]}]; data3 = Transpose[{data[[All, 1]], data[[All, 3]]}]; data4 = Transpose[{data[[All, 1]], data[[All, 4]]}]; DateListPlot[{data2, data3, data4}, Joined -> True, Filling -> {3 -> {1}}] but I have a hunch that this can be done more efficiently. I don't like the Transpose s in particular. Any ideas? edit (for extra credit) What if I need to multiply the second column by 2, which in my solution is simp...

calculus and analysis - Integral of the Sinc product

Let us consider the following integral $$ B_n = \int_0^\infty \prod_{k=1,3,5,\dots}^n\frac{\sin (x/k)}{x/k}dx $$ By definition, Sinc[x] == Sin[x]/x , therefore B[n_?OddQ] := Integrate[Product[Sinc[x/k], {k, 1, n, 2}], {x, 0, Infinity}] B /@ Range[1, 13, 2] {Pi/2, Pi/2, Pi/2, Pi/2, Pi/2, Pi/2, Pi/2} OK, everything is fine. But... B[15] 467807924713440738696537864469 Pi/935615849440640907310521750000 What's going on? Answer As Eckhard wrote in comments B[n] is the n-th Borwein integral . (The letter B was not accidental :) ) This funny properties of Borwein integrals is related to the Fourier transform of Sinc function FourierTransform[Sinc[x], x, k] 1/2 Sqrt[Pi/2] (Sign[1 - k] + Sign[1 + k]) Plot[%, {k, -2, 2}, Filling -> 0] which is the box function. The result is $\pi/2$ while the sum $$ 1/3+1/5+\dots+1/n $$ B_n = \frac{\pi}{2} - \pi \bigg(\sum_{k=3,5,\ldots}^n\frac{1}{k} -1\biggr)^\frac{n-1}{2}\prod_{k=3,5,\dots}^n\frac{k}{k-1}. $$ For $n=15$ it is equal to $$ \frac{46780...