Skip to main content

front end - How to see Monitor temporary cell with ParallelSubmit?


I have a function to load large files into a "HSQL(Standalone)" database for processing that uses Monitor to update the interface on the progress and status of the upload.


loadBigFile[t_Integer?Positive, steps_Integer?Positive] :=
 Module[{stepProgress = 0},
  Monitor[
   While[stepProgress < steps,

    (* Read batch and load to db *)
    Pause[t];
    stepProgress += 1
    ],
   Column[{StringRiffle[ToString /@ {stepProgress, steps}, " of "],
     ProgressIndicator[stepProgress, {0, steps}]},
    Alignment -> Center]
   ];
  stepProgress
  ]


loadBigFile is just to demonstrate the issue.


loadBigFile[1, 10]
(* 10 *)

enter image description here


This process takes 30+ minutes to load each file. I would like to concurrently load these files to reduce the wait time by using ParallelSubmit.


k = LaunchKernels[2];
DistributeDefinitions[loadBigFile];
{f1, f2} = {ParallelSubmit[loadBigFile[1, 5]], ParallelSubmit[loadBigFile[1, 9]]}


enter image description here


When I execute WaitAll the EvaluationObjects do run concurrently. However, instead of a monitor temporary cell for each of them I get a front end error from each kernel they are running on.


WaitAll[{f1, f2}]
CloseKernels[k];

enter image description here


(* {5, 9} *)



FrontEndObject::notavail: A front end is not available; certain operations require a front end.



How do I get the monitor temporary cells to show in the notebook for each EvaluationObject running? Or is there another method I should use for concurrent execution with Monitor?



Answer



From the link in the comment above I have devised a solution included here for ParallelSubmit specific searches.


Monitor can not be used so I make a progress function that will assist with the visual status update.


progress[current_, total_] :=
 Column[{StringRiffle[ToString /@ {current, total}, " of "],
   ProgressIndicator[current, {0, total}]},
  Alignment -> Center]


I use a status Association to hold the status updates as it makes the code more readable than using a list as with other solutions. The loadBigFile function has to change slightly.


loadBigFile[t_Integer?Positive, steps_Integer?Positive] :=
 Module[{stepProgress = 0},
  status[[Key[$KernelID]]] = <|
    "File" -> "File " <> ToString@RandomInteger[100],
    "Monitor" -> progress[stepProgress, steps]
    |>;
  While[stepProgress < steps,
   (* Read batch and load to db *)

   Pause[t];
   stepProgress += 1;

   (* Update status *)
   status[[Key[$KernelID], "Monitor"]] = progress[stepProgress, steps];
   ];

  status[[Key[$KernelID], "File"]] = status[[Key[$KernelID], "File"]] <> " done.";
  stepProgress
  ]


The Keys of the Association are the $KernelID and each has a "File" and "Monitor" key. loadBigFile updates these as it progresses.


k = LaunchKernels[2];

status = Association @@ 
     ParallelTable[$KernelID -> <|"File" -> "", "Monitor" -> ""|>, {i, $KernelCount}]
(* <|43 -> <|"File" -> "", "Monitor" -> ""|>, 42 -> <|"File" -> "", "Monitor" -> ""|>|> *)

Distribute the definitions and variables


DistributeDefinitions[progress, loadBigFile];

SetSharedVariable[status];

ParallelSubmit the jobs.


{f1, f2} = {ParallelSubmit[loadBigFile[1, 5]], ParallelSubmit[loadBigFile[1, 9]]}

enter image description here


PrintTemporary with Dynamic is used instead of Monitor. This accesses the status association to display.


PrintTemporary[
  Dynamic[Row[
    Riffle[Column[#, Alignment -> Center] & /@ 

      Query[Values, Values]@Select[#"Monitor" =!= "" &]@status, Spacer[5]]]]];
WaitAll[{f1, f2}]

enter image description here


Both have status displayed and updated concurrently.


Clean up.


UnsetShared[progress, loadBigFile, status];
CloseKernels[k];

Hope this helps.



Comments

Post a Comment

Popular posts from this blog

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...
Post a Comment
Read more

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...
Post a Comment
Read more

mathematical optimization - Minimizing using indices, error: Part::pkspec1: The expression cannot be used as a part specification

I want to use Minimize where the variables to minimize are indices pointing into an array. Here a MWE that hopefully shows what my problem is. vars = u@# & /@ Range[3]; cons = Flatten@ { Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; Minimize[{Total@((vec1[[#]] - vec2[[u[#]]])^2 & /@ Range[1, 3]), cons}, vars, Integers] The error I get: Part::pkspec1: The expression u[1] cannot be used as a part specification. >> Answer Ok, it seems that one can get around Mathematica trying to evaluate vec2[[u[1]]] too early by using the function Indexed[vec2,u[1]] . The working MWE would then look like the following: vars = u@# & /@ Range[3]; cons = Flatten@{ Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; NMinimize[ {Total@((vec1[[#]] - Indexed[vec2, u[#]])^2 & /@ R...
Post a Comment
Read more