Skip to main content

export - Exporting extremely large images; arrays generated via TensorProduct use 5 times as much RAM as expected?



I am generating plots of the roots of certain classes of polynomials, in the spirit of the multicolored visuals on the Wiki page on algebraic numbers. I've thus far been able to make some extremely spectacular plots, such as this 3600 x 3600 image of the roots of the monic quadratics and cubics (very low-res preview below)


enter image description here


and this 4600 x 4600 image of the roots of the cubics with lead coefficient 5 (low-res preview below)


enter image description here


and this 4600 x 4600 image of the roots of the cubics with lead coefficient 1 or 5 (low-res preview below).


enter image description here


I have been running into a RAM usage bottleneck during the Export step as I try to make more detailed visuals. To make the images, I essentially execute


Export["Imagename.PNG",Image[A]]

where $A$ is roughly a $4600\times 4600\times 3$ rank-3 array specifying the RGB color channels. I was having issues with the MathKernel swallowing many gigabytes of RAM and initially thought it was from the Image Export step, but it turns out it's due to the TensorProduct I used to generate $A$.



I've begun to observe that arrays generated via TensorProduct occupy 5 times as much RAM as I would naively expect. For example,


$HistoryLength = 0;
L = 2000;
a = RandomReal[{-10^6, 10^6}, {L, L}];
b = RandomReal[{-10^6, 10^6}, 3];
c = TensorProduct[a, b];
d = RandomReal[{-10^6, 10^6}, {L, L, 3}];
ByteCount[a]
ByteCount[b]
ByteCount[c]

ByteCount[d]
Quit[]

indicates that $c$ occupies 480 MB RAM, while $d$ occupies 96MB RAM. Choosing $L=4600$ shows that $c$ occupies 2.6 GB RAM, while $d$ occupies 500 MB RAM. Changing the ranges of RandomReal from $10^6$ to other numbers does not significantly alter the result. This is a bit unexpected, since both $c$ and $d$ are of the same dimension and same MachinePrecision entry type, and yet the manner of array generation seems to strongly impact the space it occupies in RAM.


Question: Does anyone know why using TensorProduct to generate an array creates an array object which occupies 5 times the normal RAM amount as comparable arrays generated via other means?


EDIT: After applying the packed array fixes, things work well. Here are links to the two notebooks that can be used to generate the images I used. Root Generator is used to compute and export the polynomial roots of one's choice to sparse matrix formats, and Root Visualizer imports the sparse matrices and exports the images. I use packed arrays in the RAM-intensive parts, and Compile the convolution kernel used to blur the image. If anyone sees any obvious improvements that could be made, I'd love to hear it.



Mathematica 10 has updated TensorProduct so that it returns packed arrays. So if you are running version 10 or higher, you should not run into this problem.



Answer



For unknown reasons, TensorProduct produces unpacked array (see packed arrays here). You can use Outer[Times, a, b] instead:



$HistoryLength = 0;
L = 2000;
a = RandomReal[{-10^6, 10^6}, {L, L}];
b = RandomReal[{-10^6, 10^6}, 3];
c = TensorProduct[a, b];
c2 = Outer[Times, a, b];
d = RandomReal[{-10^6, 10^6}, {L, L, 3}];
ByteCount /@ {a, b, c, c2, d} // Column
Developer`PackedArrayQ /@ {c, c2} // Column
Max@Abs[c - c2]



32000152
128
480800392
96000160
96000160

False
True


0.

Comments

Popular posts from this blog

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...