Skip to main content

parallelization - Effective parallel processing of large items


When trying to parallelize a task which involves items of larger sizes, I cannot achieve efficient parallelization. To demonstrate this, we launch some kernels:


LaunchKernels[12];


I am facing this problem with real-world data which I cannot share here. Instead, we generate some random data and adjust the following parameters to a) reflect the problem and b) get some reasonable timings:


itemsize = 20000; (* The size of an individual item *)
numberofitems = 200; (* Number of items to process *)
difficulty = 500;(* Processing diffculty: the part of an individual item that is actually processed *)

Now let's generate the random values:


randomValues = Parallelize@Table[
Transpose@{RandomReal[1, itemsize], RandomReal[1, itemsize]},
{numberofitems}
];


An individual item has 320 kB, the full dataset is 64 MB in size:


ByteCount /@ {randomValues[[1]], randomValues}
(* {320152, 64032072} *)

Now we compare Map and ParallelMap with an arbitrary function that takes a reasonable amount of time to process, like FindCurvePath.


map = Map[
FindCurvePath[#[[1 ;; difficulty]]] &,
randomValues
]; // AbsoluteTiming


(* {11.9619, Null} *)

pmap = ParallelMap[
FindCurvePath[#[[1 ;; difficulty]]] &,
randomValues,
Method -> "ItemsPerEvaluation" -> 10
]; // AbsoluteTiming

(* {23.6492, Null} *)


Surprisingly, the parallel version is twice as slow. When watching the CPU usage of the main kernel vs. the subkernels, it is notable that most of the evaluation time is spent at the beginning in the main kernel. Then, processing in the subkernels is done in less than 2 seconds.


Note that I intentionally made the items larger than what is actually processed, so the full items of 320 kB in size (20 000 * 20 000 random reals) need to be distributed to the subkernels. If we reduce the item size to the amount that is actually processed, things change drastically:


pmap2 = ParallelMap[
FindCurvePath[#[[1 ;; difficulty]]] &,
randomValues[[All, 1 ;; difficulty]], (* only use a small part of the items *)
Method -> "ItemsPerEvaluation" -> 10
]; // AbsoluteTiming

(* {2.03152, Null} *)


Now we get a performance improvement as expected. The result is the same:


map === pmap === pmap2
(* True *)

Apparently, the distribution of the large items to the subkernels is the bottleneck. Note that unlike in this demonstration, my real-world application does need all the data that is present in the items.


I did not find any way to improve the parallel performance. Changing the method to FinestGrained or CorsestGrained performs worse. Any ideas how to make parallel processing efficient?



Answer



Seralization of the data to a ByteArray object seems to overcome the data transfer bottleneck. The necessary functions BinarySerialize and BinaryDeserialize have been introduced in 11.1.


Here is a simple function implementing a ParallelMap which serializes the data before the transfer to the subkernels and makes the subkernels deseralize it before processing:



ParallelMapSerialized[f_, data_, opts___] := ParallelMap[
f[BinaryDeserialize@#] &,
BinarySerialize /@ data,
opts
]

Running the benchmark again:


map = Map[
FindCurvePath[#[[1 ;; difficulty]]] &,
randomValues

]; // AbsoluteTiming

(* {9.60715, Null} *)

pmap = ParallelMap[
FindCurvePath[#[[1 ;; difficulty]]] &,
randomValues,
Method -> "ItemsPerEvaluation" -> 10
]; // AbsoluteTiming


(* {17.5937, Null} *)

pmapserialized = ParallelMapSerialized[
FindCurvePath[#[[1 ;; difficulty]]] &,
randomValues,
Method -> "ItemsPerEvaluation" -> 10
]; // AbsoluteTiming

(* {1.85387, Null} *)


pmap === pmap2 === pmapserialized
(* True *)

Serialization led to a performance increase of almost 10-fold compared to ParallelMap, and to a 5-fold increase compared to serial processing.


Comments

Popular posts from this blog

plotting - How to draw lines between specified dots on ListPlot?

I would like to create a plot where I have unconnected dots and some connected. So far, I have figured out how to draw the dots. My code is the following: ListPlot[{{1, 1}, {2, 2}, {3, 3}, {4, 4}, {1, 4}, {2, 5}, {3, 6}, {4, 7}, {1, 7}, {2, 8}, {3, 9}, {4, 10}, {1, 10}, {2, 11}, {3, 12}, {4,13}, {2.5, 7}}, Ticks -> {{1, 2, 3, 4}, None}, AxesStyle -> Thin, TicksStyle -> Directive[Black, Bold, 12], Mesh -> Full] I have thought using ListLinePlot command, but I don't know how to specify to the command to draw only selected lines between the dots. Do have any suggestions/hints on how to do that? Thank you. Answer One possibility would be to use Epilog with Line : ListPlot[ {{1, 1}, {2, 2}, {3, 3}, {4, 4}, {1, 4}, {2, 5}, {3, 6}, {4, 7}, {1, 7}, {2, 8}, {3, 9}, {4, 10}, {1, 10}, {2, 11}, {3, 12}, {4, 13}, {2.5, 7}}, Ticks -> {{1, 2, 3, 4}, None}, AxesStyle -> Thin, TicksStyle -> Directive[Black, Bold, 12], Mesh -> Full, Epilog -> { Line[ ...

calculus and analysis - Integral of the Sinc product

Let us consider the following integral $$ B_n = \int_0^\infty \prod_{k=1,3,5,\dots}^n\frac{\sin (x/k)}{x/k}dx $$ By definition, Sinc[x] == Sin[x]/x , therefore B[n_?OddQ] := Integrate[Product[Sinc[x/k], {k, 1, n, 2}], {x, 0, Infinity}] B /@ Range[1, 13, 2] {Pi/2, Pi/2, Pi/2, Pi/2, Pi/2, Pi/2, Pi/2} OK, everything is fine. But... B[15] 467807924713440738696537864469 Pi/935615849440640907310521750000 What's going on? Answer As Eckhard wrote in comments B[n] is the n-th Borwein integral . (The letter B was not accidental :) ) This funny properties of Borwein integrals is related to the Fourier transform of Sinc function FourierTransform[Sinc[x], x, k] 1/2 Sqrt[Pi/2] (Sign[1 - k] + Sign[1 + k]) Plot[%, {k, -2, 2}, Filling -> 0] which is the box function. The result is $\pi/2$ while the sum $$ 1/3+1/5+\dots+1/n $$ B_n = \frac{\pi}{2} - \pi \bigg(\sum_{k=3,5,\ldots}^n\frac{1}{k} -1\biggr)^\frac{n-1}{2}\prod_{k=3,5,\dots}^n\frac{k}{k-1}. $$ For $n=15$ it is equal to $$ \frac{46780...

list manipulation - Selecting multiple columns from a matrix?

Sample data: data = { {{2013, 1, 1}, 24.13, 167.67, 231.82}, {{2013, 1, 2}, 32.15, 170.92, 225.99}, {{2013, 1, 3}, 35.43, 172.68, 221.67}, {{2013, 1, 4}, 36.73, 173.05, 218.32}, {{2013, 1, 5}, 58.19, 165.96, 197.05}, {{2013, 1, 6}, 69.99, 163.50, 187.52}, {{2013, 1, 7}, 71.37, 154.21, 175.58}, {{2013, 1, 8}, 72.51, 149.66, 163.25}}; I want a DateListPlot with three graphs, so for a matrix formed by columns 1 and 2, one for columns 1 and 3, and 1 for columns 1 and 4. At the moment I'm using this code: data2 = Transpose[{data[[All, 1]], data[[All, 2]]}]; data3 = Transpose[{data[[All, 1]], data[[All, 3]]}]; data4 = Transpose[{data[[All, 1]], data[[All, 4]]}]; DateListPlot[{data2, data3, data4}, Joined -> True, Filling -> {3 -> {1}}] but I have a hunch that this can be done more efficiently. I don't like the Transpose s in particular. Any ideas? edit (for extra credit) What if I need to multiply the second column by 2, which in my solution is simp...