Skip to main content

performance tuning - Launching remote kernels - will it help RAM needs?


Given the exasperating RAM needs of my Mathematica program, I contemplated launching remote kernels to run my Mathematica script. This is a two part question:


Part 1


If I launch a remote kernel(s), is the need for RAM also distributed amongst all the remote computers?


Part 2


I keep getting an error message when I try to launch a remote kernel from the Mathematica GUI.



The kernel c20-0707-23 failed to connect to the front end. (Error = MLECONNECT). You should try running the kernel connection outside the front end.



I have located the Kernel Program correctly on the remote computer. When I try starting a kernel over ssh in a terminal window with:



ssh username@remote.computer /path/to/math

Or


ssh username@remote.computer /path/to/MathKernel

After I enter the password to log in to the remote computer, I see that a kernel has been launched.


However, the same doesn't occur with the front end.


Are there any easy to understand tutorials/examples on launching remote kernels, without needing the GUI as I'd like to run scripts? I tried reading, this, for instance and the only thing that it did was make me panic as it was greek. I also read this but it's not an entirely similar problem


Here's a sample script that I used to launch local kernels and execute a simple NDSolve command:


#!/usr/local/bin/MathematicaScript -script


CloseKernels[]
LaunchKernels[3]
Print["AbsoluteTiming reveals:" NDSolve[{y'[x] == y[x] Cos[x + y[x]],
     y[0] == 1}, y, {x, 0, 30}]; // AbsoluteTiming ]
CloseKernels[]

This question may help answer the other question I linked in the first line.



Answer



Sometimes, yes it is sometimes distributed. I can't say for sure when and why, but I have noticed cases where ParallelMap does not distribute all the memory, but ParallelTable does.



Basically, Mathematica is trying to make things easy. You don't always want to have to spell out each and every variable that you want available on a remote kernel, and so it does its best to figure out just what you need. There are lots of options in Mathematica 8.


If you're trying to "aggregate" memory on other nodes, then you have to be careful to create the large data on remote machines to avoid the network transfer of the contents of the master kernel to the other machines. (Please know that I have run into issues in the past, where there seemed to be a limit as to precisely how much memory could be allocated by a Remote Kernel itself. i.e. couldn't allocate a gigabyte, let alone 16 gigabytes that are found in machines today. I don't know if this is still a problem.)


Comments

Post a Comment

Popular posts from this blog

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...
Post a Comment
Read more

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...
Post a Comment
Read more