parallelization - Writing data to a common file during parallel processing

I am executing a loop, and in each loop I have to store a parameter in a file. It is like

file = FileNameJoin[{NotebookDirectory[], "info.txt"}];

OpenWrite[file];
Do[out = Prime[i]; WriteString[file, i, "\t", out, "\n"], {i, 1, 10}]
Close[file]

It stores the values perfectly. But when I use Parallelize[], it produce only partial output.

file = FileNameJoin[{NotebookDirectory[], "infop.txt"}];
OpenWrite[file];
Do[out = Prime[i]; WriteString[file, i, "\t", out, "\n"], {i, 1, 10}]
Close[file]

It produces only 4-5 sets out of 10 (They are in random ordering, but that is not a problem). Although when I use Print[], it shows all the sets.

Parallelize[Do[out = Prime[i]; Print[i , "\t", out], {i, 1, 10}]]

The loop I am using for work is pretty big and I may have to check the info.txt file several times to know the status while the code is running. Is there any way to write the data (as the code is running) in a common file while using parallel processing.

Answer

Interaction between parallel kernels

This is a common parallelisation problem. Without proper blocking you'll produce such results, but with blocking you loose all your parallelisation. So you've to come up with a solution with at least interaction between the parallel sub-kernels as possible. The less interaction, the faster things will go.

A solution is to use a shared variable to communicate results found to the master kernel:

file = FileNameJoin[{$TemporaryDirectory, "info.txt"}];
OpenWrite[file];


SetSharedVariable[list]
list = {};
ParallelDo[out = Prime[i]; AppendTo[list, {i, out}], {i, 1, 10}]

list = Sort[list]; (* only if you care about order *)

WriteString[file, #[[1]], "\t", #[[2]], "\n"] & /@ list;

Close[file]

If you want to, you could aggregate this into a function, but then you've to take care about the distributed definition of your function to the parallel kernels. Not if you define your function in the current context, because those are distributed automatically.

Edit:

I do now parallelisation for years (not necessarily in Mma) and suffered a lot to get things real going, but what I learned from my experiences is:

1. Explore all serial ways to speed things up, before you go parallel


2. If you go parallel beware of synchronisation, because this is killing parallelism

Edit 2:

This is maybe not the most elegant solution to merge files in a directory, but it does the job. Imagine you have several files with data in it in a flat directory. For instance:

• /tmp/test/test1.txt -> "Hello"


• /tmp/test/test2.txt -> "World"

You can read all the files in the directory into a list:

strList = Read /@ OpenRead /@ (SetDirectory["/tmp/test"]; 
    FileNameJoin[{"/tmp/test", #}] & /@ FileNames[]);

And now you can OpenWrite and write to that channel mapping over strList and you're done.

This is all, if you don't want to leave your tool (like me; probably emacs affected ;) )