Skip to main content

export - Exporting extremely large images; arrays generated via TensorProduct use 5 times as much RAM as expected?



I am generating plots of the roots of certain classes of polynomials, in the spirit of the multicolored visuals on the Wiki page on algebraic numbers. I've thus far been able to make some extremely spectacular plots, such as this 3600 x 3600 image of the roots of the monic quadratics and cubics (very low-res preview below)


enter image description here


and this 4600 x 4600 image of the roots of the cubics with lead coefficient 5 (low-res preview below)


enter image description here


and this 4600 x 4600 image of the roots of the cubics with lead coefficient 1 or 5 (low-res preview below).


enter image description here


I have been running into a RAM usage bottleneck during the Export step as I try to make more detailed visuals. To make the images, I essentially execute


Export["Imagename.PNG",Image[A]]

where $A$ is roughly a $4600\times 4600\times 3$ rank-3 array specifying the RGB color channels. I was having issues with the MathKernel swallowing many gigabytes of RAM and initially thought it was from the Image Export step, but it turns out it's due to the TensorProduct I used to generate $A$.



I've begun to observe that arrays generated via TensorProduct occupy 5 times as much RAM as I would naively expect. For example,


$HistoryLength = 0;
L = 2000;
a = RandomReal[{-10^6, 10^6}, {L, L}];
b = RandomReal[{-10^6, 10^6}, 3];
c = TensorProduct[a, b];
d = RandomReal[{-10^6, 10^6}, {L, L, 3}];
ByteCount[a]
ByteCount[b]
ByteCount[c]

ByteCount[d]
Quit[]

indicates that $c$ occupies 480 MB RAM, while $d$ occupies 96MB RAM. Choosing $L=4600$ shows that $c$ occupies 2.6 GB RAM, while $d$ occupies 500 MB RAM. Changing the ranges of RandomReal from $10^6$ to other numbers does not significantly alter the result. This is a bit unexpected, since both $c$ and $d$ are of the same dimension and same MachinePrecision entry type, and yet the manner of array generation seems to strongly impact the space it occupies in RAM.


Question: Does anyone know why using TensorProduct to generate an array creates an array object which occupies 5 times the normal RAM amount as comparable arrays generated via other means?


EDIT: After applying the packed array fixes, things work well. Here are links to the two notebooks that can be used to generate the images I used. Root Generator is used to compute and export the polynomial roots of one's choice to sparse matrix formats, and Root Visualizer imports the sparse matrices and exports the images. I use packed arrays in the RAM-intensive parts, and Compile the convolution kernel used to blur the image. If anyone sees any obvious improvements that could be made, I'd love to hear it.



Mathematica 10 has updated TensorProduct so that it returns packed arrays. So if you are running version 10 or higher, you should not run into this problem.



Answer



For unknown reasons, TensorProduct produces unpacked array (see packed arrays here). You can use Outer[Times, a, b] instead:



$HistoryLength = 0;
L = 2000;
a = RandomReal[{-10^6, 10^6}, {L, L}];
b = RandomReal[{-10^6, 10^6}, 3];
c = TensorProduct[a, b];
c2 = Outer[Times, a, b];
d = RandomReal[{-10^6, 10^6}, {L, L, 3}];
ByteCount /@ {a, b, c, c2, d} // Column
Developer`PackedArrayQ /@ {c, c2} // Column
Max@Abs[c - c2]



32000152
128
480800392
96000160
96000160

False
True


0.

Comments

Popular posts from this blog

plotting - How to draw lines between specified dots on ListPlot?

I would like to create a plot where I have unconnected dots and some connected. So far, I have figured out how to draw the dots. My code is the following: ListPlot[{{1, 1}, {2, 2}, {3, 3}, {4, 4}, {1, 4}, {2, 5}, {3, 6}, {4, 7}, {1, 7}, {2, 8}, {3, 9}, {4, 10}, {1, 10}, {2, 11}, {3, 12}, {4,13}, {2.5, 7}}, Ticks -> {{1, 2, 3, 4}, None}, AxesStyle -> Thin, TicksStyle -> Directive[Black, Bold, 12], Mesh -> Full] I have thought using ListLinePlot command, but I don't know how to specify to the command to draw only selected lines between the dots. Do have any suggestions/hints on how to do that? Thank you. Answer One possibility would be to use Epilog with Line : ListPlot[ {{1, 1}, {2, 2}, {3, 3}, {4, 4}, {1, 4}, {2, 5}, {3, 6}, {4, 7}, {1, 7}, {2, 8}, {3, 9}, {4, 10}, {1, 10}, {2, 11}, {3, 12}, {4, 13}, {2.5, 7}}, Ticks -> {{1, 2, 3, 4}, None}, AxesStyle -> Thin, TicksStyle -> Directive[Black, Bold, 12], Mesh -> Full, Epilog -> { Line[ ...

dynamic - How can I make a clickable ArrayPlot that returns input?

I would like to create a dynamic ArrayPlot so that the rectangles, when clicked, provide the input. Can I use ArrayPlot for this? Or is there something else I should have to use? Answer ArrayPlot is much more than just a simple array like Grid : it represents a ranged 2D dataset, and its visualization can be finetuned by options like DataReversed and DataRange . These features make it quite complicated to reproduce the same layout and order with Grid . Here I offer AnnotatedArrayPlot which comes in handy when your dataset is more than just a flat 2D array. The dynamic interface allows highlighting individual cells and possibly interacting with them. AnnotatedArrayPlot works the same way as ArrayPlot and accepts the same options plus Enabled , HighlightCoordinates , HighlightStyle and HighlightElementFunction . data = {{Missing["HasSomeMoreData"], GrayLevel[ 1], {RGBColor[0, 1, 1], RGBColor[0, 0, 1], GrayLevel[1]}, RGBColor[0, 1, 0]}, {GrayLevel[0], GrayLevel...

Is there a way to do conditional matrix loop using 'continue'

I have the following: n = 3; m = 5; ww = RandomReal[{0, 0.1}, {n, n}]; uu = RandomReal[{0, 1}, {m, n}]; pp = RandomReal[{0, 1}, {n, n}]; ss = RandomInteger[{0, 5}, {m, n}]; Grid[{{"ww", "uu", "pp", "ss"}, {ww // TableForm, uu // TableForm, pp // TableForm, ss // TableForm}}, Spacings -> {5, 2}, Dividers -> All] where I would like to look at every element of matrix ss and produce a matrix tt , with zeroes at the locations in ss which have zeroes, and in all other positions do the following: tt = (-1/Subscript[ww, m]) Log[(1 - uu)/(Subscript[pp, m - 1])], where Subscript[ww, m] is the value at index of ww matrix and where Subscript[pp, m - 1] is the value at index-1 of pp matrix. So for example if the first value ever read from matrix ss happens to be 2, then value taken from matrix ww would be from the row 2, but from pp would be from row 1. Also how to tell difference between a 0 as a valid value from within the matrix elemen...