Skip to main content

front end - How to see Monitor temporary cell with ParallelSubmit?


I have a function to load large files into a "HSQL(Standalone)" database for processing that uses Monitor to update the interface on the progress and status of the upload.


loadBigFile[t_Integer?Positive, steps_Integer?Positive] :=
 Module[{stepProgress = 0},
  Monitor[
   While[stepProgress < steps,

    (* Read batch and load to db *)
    Pause[t];
    stepProgress += 1
    ],
   Column[{StringRiffle[ToString /@ {stepProgress, steps}, " of "],
     ProgressIndicator[stepProgress, {0, steps}]},
    Alignment -> Center]
   ];
  stepProgress
  ]


loadBigFile is just to demonstrate the issue.


loadBigFile[1, 10]
(* 10 *)

enter image description here


This process takes 30+ minutes to load each file. I would like to concurrently load these files to reduce the wait time by using ParallelSubmit.


k = LaunchKernels[2];
DistributeDefinitions[loadBigFile];
{f1, f2} = {ParallelSubmit[loadBigFile[1, 5]], ParallelSubmit[loadBigFile[1, 9]]}


enter image description here


When I execute WaitAll the EvaluationObjects do run concurrently. However, instead of a monitor temporary cell for each of them I get a front end error from each kernel they are running on.


WaitAll[{f1, f2}]
CloseKernels[k];

enter image description here


(* {5, 9} *)



FrontEndObject::notavail: A front end is not available; certain operations require a front end.



How do I get the monitor temporary cells to show in the notebook for each EvaluationObject running? Or is there another method I should use for concurrent execution with Monitor?



Answer



From the link in the comment above I have devised a solution included here for ParallelSubmit specific searches.


Monitor can not be used so I make a progress function that will assist with the visual status update.


progress[current_, total_] :=
 Column[{StringRiffle[ToString /@ {current, total}, " of "],
   ProgressIndicator[current, {0, total}]},
  Alignment -> Center]


I use a status Association to hold the status updates as it makes the code more readable than using a list as with other solutions. The loadBigFile function has to change slightly.


loadBigFile[t_Integer?Positive, steps_Integer?Positive] :=
 Module[{stepProgress = 0},
  status[[Key[$KernelID]]] = <|
    "File" -> "File " <> ToString@RandomInteger[100],
    "Monitor" -> progress[stepProgress, steps]
    |>;
  While[stepProgress < steps,
   (* Read batch and load to db *)

   Pause[t];
   stepProgress += 1;

   (* Update status *)
   status[[Key[$KernelID], "Monitor"]] = progress[stepProgress, steps];
   ];

  status[[Key[$KernelID], "File"]] = status[[Key[$KernelID], "File"]] <> " done.";
  stepProgress
  ]


The Keys of the Association are the $KernelID and each has a "File" and "Monitor" key. loadBigFile updates these as it progresses.


k = LaunchKernels[2];

status = Association @@ 
     ParallelTable[$KernelID -> <|"File" -> "", "Monitor" -> ""|>, {i, $KernelCount}]
(* <|43 -> <|"File" -> "", "Monitor" -> ""|>, 42 -> <|"File" -> "", "Monitor" -> ""|>|> *)

Distribute the definitions and variables


DistributeDefinitions[progress, loadBigFile];

SetSharedVariable[status];

ParallelSubmit the jobs.


{f1, f2} = {ParallelSubmit[loadBigFile[1, 5]], ParallelSubmit[loadBigFile[1, 9]]}

enter image description here


PrintTemporary with Dynamic is used instead of Monitor. This accesses the status association to display.


PrintTemporary[
  Dynamic[Row[
    Riffle[Column[#, Alignment -> Center] & /@ 

      Query[Values, Values]@Select[#"Monitor" =!= "" &]@status, Spacer[5]]]]];
WaitAll[{f1, f2}]

enter image description here


Both have status displayed and updated concurrently.


Clean up.


UnsetShared[progress, loadBigFile, status];
CloseKernels[k];

Hope this helps.



Comments

Post a Comment

Popular posts from this blog

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...
Post a Comment
Read more

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...
Post a Comment
Read more

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more