kernel - Saving data inside a notebook so that I don't have to run it again?

Piggybacking on this, I am somehow not fully convinced that I can't save data generated by a calculation in a mathematica file so that when I re-launch said file, I wouldn't have to run my calculations again.

I ask because I use NDSolve for 4th order non linear PDEs and sometimes I need to run it for really large times (in excess of a few hours, yes that sounds crazy but I am just trying to get results for a fluid dynamics problem here and I am not all that interested in being a computer scientist to reduce run times).

So after reading the linked article, I did this for an example problem:

sol = NDSolve[{D[u[t, x], t] == D[u[t, x], x, x], u[0, x] == 0, 
   u[t, 0] == Sin[t], u[t, 5] == 0}, u, {t, 0, 10}, {x, 0, 5}]

DumpSave["pde0.mx", sol]

Then I quit mathematica and relaunch it and load pde0.mx with Get

Get["pde0.mx"]

And when I plot,

Plot3D[Evaluate[u[t, x] /. sol], {t, 0, 10}, {x, 0, 5}, 
 PlotRange -> All]

Voila, I get the plot as if I had run the simulation.

So.. did I run the simulation again by invoking sol through Get or was my kernel state saved?

Answer

What was saved was the content of sol, which happens to contain the solution to your equation (you explicitly set it to that), and therefore is certainly sufficient for your plot.

Saving Kernel state however would involve saving things like the random seed, so the following would give the same output twice (using a hypothetical function SaveKernelState and corresponding LoadKernelState):

SaveKernelState["somefile"]
Print[RandomInteger[10, 10]]
(*
==> {5, 0, 1, 1, 7, 4, 4, 7, 9, 8}
*)
LoadKernelState["somefile"]
Print[RandomInteger[10, 10]]
(*

==> {5, 0, 1, 1, 7, 4, 4, 7, 9, 8}
*)

Also there are internal caches for things like FullSimplify, e.g.

FullSimplify[Sum[Sin[k^2 x],{k,0,8}]]//Timing       
(*
{2.89218, Sin[x] + Sin[4 x] + Sin[9 x] + Sin[16 x] + Sin[25 x] + 
Sin[36 x] + Sin[49 x] + Sin[64 x]}
*)
FullSimplify[Log[Sum[Sin[k^2 x],{k,0,8}]]]//Timing 

(*
{0.028002, Log[Sin[x] + Sin[4 x] + Sin[9 x] + Sin[16 x] + Sin[25 x] + 
Sin[36 x] + Sin[49 x] + Sin[64 x]]}
*)

Here, the result of the first FullSimplify was cached and reused in the second one. Saving full Kernel state would include saving those caches, so the second simplification would go much faster in the restarted session as well.

Edit: After reading Leonid's answer and the comments (as well as the answer he linked), I've now written a function which does exactly what you ask for in your title: Save the data inside your notebook:

SetAttributes[PermanentSet,HoldAll];
PermanentSet[var_Symbol,value_]:=
  (If[OwnValues[var] === {},

      Module[{nb=EvaluationNotebook[]},
             SelectionMove[nb, Before, EvaluationCell];
             NotebookWrite[nb,Cell[ToString[Unevaluated[var]] <>
                                   " = Uncompress[\"" <>
                                   Compress[var=value] <>
                                   "\"]",
                                   "Input",
                                   Editable->False]]]];
   var)

This is used as follows: To set the (previously unassigned) variable a to the result of the time-consuming calculation Pause[2];1+1, just write

PermanentSet[a, Pause[2];1+1]

Executing this while a is not set will evaluate the expression and assign the result (2) to a, but will additionally add a cell before this one, containing

a = Uncompress["1:eJxTTMoPymRiYGAAAAtMAbA="]

(now in this case, a = 2 would have been shorter :-)). So when you evaluate the notebook in order, you'll first evaluate that line, setting a to 2, and only then the PermanentSet. Since PermanentSet now finds a already assigned, it doesn't evaluate the second argument again, but just returns the value.

Bugs and Limitations:

• If the evaluation contains side effects, those side effects will not be executed when the variable is set from the previous cell. Therefore this should only be used for side-effect-free calculations.



• This code depends on the previous cell being evaluated before this one. Otherwise the evaluation is restarted. However, with side-effect-free calculations, it should be safe to abort that calculation and execute the previous cell.


• This code doesn't work if the variable has a value, therefore if you want to replace an existing value, you have to unset the value first. However that unsetting must not happen in the same cell, but in a cell before. Otherwise it will undo the setting by the previous cell on re-evaluation.


• If the cell containing PermanentSet is an initialization cell, the generated cell should be an initialization cell as well. However it isn't (because I don't know how to do that).


• Also, this will place the selection immediately before the cell containing the call. Ideally it would save where the current selection is and restore it afterwards. However I don't know how to do that either.

Edit 2:

Based on Rojo's idea to store the data in notebook tagging rules (a feature which I didn't know about before), I've now written a different version of PermanentSet which resolves some of the problems with the previous one. It now saves both the expression and the resulting value in the tagging rules. This way, the function is evaluated again iff the expression has changed.

SetAttributes[PermanentSet,HoldAll];
PermanentSet[var_Symbol, value_] :=
  Module[{nb=EvaluationNotebook[],

          name=ToString[Unevaluated[var]],
          expr=Compress[Unevaluated[value]]},
    If[TrueQ[CurrentValue[nb, {TaggingRules, "Storage",
                               name <> "expression"}] == expr],
      var = Uncompress@CurrentValue[nb, {TaggingRules, "Storage", name<>"value"}],
      CurrentValue[nb, {TaggingRules, "Storage", name<>"expression"}] = expr;
      CurrentValue[nb,{TaggingRules, "Storage", name<>"value"}] =
        Compress[var=value]];
    var];

Bugs and Limitations:

• As soon as the expression is evaluated, it won't be evaluated again until the expression is changed. That may be desired, but it also may be undesired (e.g. if re-evaluating because some variables changed). You can force a re-evaluation be PermanentSetting to a dummy value (e.g. Null) before re-evaluating. I don't see an easy way to automate that, though, because the variables may be used in functions called during the evaluation; therefore it's not possible to automatically determine which variables/values are needed. Maybe a TrackedVariables option like the one for DynamicModule would be useful for this.


• There probably should be a PermanentUnset counterpart to PermanentSet.


• As in the previous version, it is not a good idea to have side effects in the expression, because they will not happen again.