Skip to main content

memory - How to force Mathematica to clean up the cache



(see the bottom of this question for the MWE)


I'm working with a set of numerical data: numerous (a few thousands) files in the style of list of vectors (quadruplets x, y, vx, vy). Files are named as v1.txt, v2.txt, ..., vk.txt, where k is a few thousands. I am using For loop in Mathematica, in each cycle several steps are performed:


1) load vi.txt
2) make list of pairs of pairs from it (in the form of {{x1, y1},{v1x, v1y}, ...})
3) construct absolute value of the vector field ({{x1, y1}, norm of {v1x, v1y}, ...})
4) Make list density plot of the absolute value list
5) Make list stream plot of the pairs of pairs
6) export both.

The problem is that after about 200 cycles, Mathematica sucks up all the physical memory available in my machine and the kernel crashes. I tried using:



1) $HistoryLength = 0
2) ClearSystemCache[] after every cycle
3) ClearAll[(plots, data)] after every cycle
4) all of the above combined

up to no avail. What am I doing wrong? What should I do to tell Mathematica to get rid of the images from the previous step, so it won't suck up all of my RAM? The code I'm using is as follows:


$HistoryLength = 0;
SetDirectory[NotebookDirectory[]];
Monitor[
For[i = 1, i <= 5383, i++,

v = Partition[
Partition[
Flatten@Import["output\\results\\v" <> ToString[i] <> ".txt",
"Table"], 2], 2];
Absv = Table[{v[[i, 1, 1]], v[[i, 1, 2]], Norm[v[[i, 2]]]}, {i, 1,
Length@v}];
LDP = ListDensityPlot[Absv, PlotRange -> Full,
ColorFunction -> ColorData[{"SolarColors", "Reverse"}],
MaxPlotPoints -> 100, ImageSize -> 1000];
LSP = ListStreamPlot[v, StreamStyle -> White, StreamPoints -> Fine,

StreamScale -> 0.1, ImageSize -> 1000];
Export["output\\results\\img" <> ToString[i] <> ".png", LDP];
Export["output\\results\\img_stream" <> ToString[i] <> ".png", Show[LDP, LSP]];
ClearSystemCache[];
ClearAll[v,Absv,LDP,LSP];
], i]

Why is Mathematica completely ignoring all statements $HistoryLength, ClearSystemCache and ClearAll?


MWE


This is a minimal working example demonstrating my problem (not depending on any mysterious data that I have on my HDD):



MemoryInUse[]
Do[
ListStreamPlot@
Table[{{Random[], Random[]}, {Random[], Random[]}}, 1000];
ClearSystemCache[],
{i, 10}
];
MemoryInUse[]

Together with Henrik we figured out ListStreamPlot is the culprit, therefore I reported this as a bug.




Answer



This is also not an answer (I think you found the culprit and it needs to be solved by WRI) but a suggestion for a workaround. The idea is to run the memory leaking code in an extra kernel and restart that kernel every once in a while. Using the parallel functionality of current Mathematica releases this is pretty simple and by choosing an appropriate value for the number of parallel kernels you could even take advantage of parallelism if desired.


Here is a simple example based on your minimal working example which demonstrates the idea and shows that the master kernel will not accumulate any memory:


LaunchKernels[1];
Print[MemoryInUse[]];
calcsPerKernel = 3;
Do[
With[{fname = "plot-" <> ToString[i] <> ".png"},
ParallelEvaluate[
Export[

FileNameJoin[{$HomeDirectory, "Desktop", fname}],
ListStreamPlot[
Table[{{Random[], Random[]}, {Random[], Random[]}}, 1000]
]
];
Print[MemoryInUse[]];
]
];
If[Mod[i, calcsPerKernel] == 0,
CloseKernels[];

ClearSystemCache[];
LaunchKernels[1];
];
,
{i, 10}
];
CloseKernels[];
ClearSystemCache[];
Print[MemoryInUse[]];


The idea for a more general and flexible approach is as follows:



  • use LaunchKernels to launch the desired number of parallel kernels

  • use ParallelSubmit to pass the computations (e.g. per file) to the parallel kernel(s)

  • use WaitNext to get the result of a computation from the other kernel(s).

  • use CloseKernels[kernelid] and LaunchKernels to restart a/the parallel kernels


As it is very difficult to completely avoid memory leaks in sufficient complex Mathematica code I think this is a technique that will be helpful in many situations. We are using this approach regularly for computations that run for several days and have found that it works very reliable. Unlike some other parts of Mathematica the parallel toolkit seems to work very stable and reliable even on a larger scale.


Comments

Popular posts from this blog

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...