Skip to main content

import - Why does ImageData need four times more memory?


Background


We are dealing with very large TIFF image files that are imported, processed and exported using Mathematica 9.0.1. Many of our algorithms only work with an array representation of the images, where pixel values are stored in the form of nested lists. For this, we are currently using the function ImageData to directly convert images into an array representation.


Problem


I am struggeling with the increase in memory usage when converting images to the array representation using ImageData. What I am wondering about is that when I convert images with ImageData the allocated memory is four times higher compared to having only the images in memory.


Example


Consider the following example: I first create a stack of 50 images that is exported to a TIFF file. Then I import the file again with the option "Data". I check the memory before and after Import. Compared to importing the file without any option, this takes up four times more memory.



Export["testFile.tif", ConstantArray[Image[ConstantArray[0, {1024, 1024}], "Bit16"], 50]];
FileByteCount["testFile.tif"]


104901507



MemoryInUse[]


8617736552




imgData = Import["testFile.tif", "Data"];
MemoryInUse[]


9037228760



9037228760 - 8617736552



419492208



If I import the file without the option, the memory used is about 104905128 bytes. This gives a factor of about 4 (3.99) more memory needed. I repeated this several times and I have to admit I do not have any explanation.


Questions



  1. Is there a simple explanation for the increase of memory usage?

  2. How can I reduce the amount of allocated memory when using the array representation?


Attempts


After reading about the function DeveloperToPackedArray` in What is a Mathematica packed array? I tried to convert the data array hoping that this would give me a decrease in memory usage. Nevertheless, I am not really sure if I am using the function correctly in this case. The code I tried is the following:



packed = Developer`ToPackedArray[Import["testFile.tif", "Data"]];

System specifictation



  • OS: Windows 7 Professional, Service Pack 1 (64-bit)

  • Processor: Intel Xeon CPU E5630 @ 2.53 GHz (2 Processors)

  • Installed memory: 96 GB

  • Mathematica 9.0.1

  • Java version 1.7.0_07



Additional question


Still one problem remains: Is their a clean way in Mathematica, how I could reduce the number of bits used for Integer to change the amount of memory used for the array representation (ImageData) without losing information? Or do I have to stick to Image in that case?



Answer



On my system (Windows 7 64-bit, 12GB, Mathematica v8) I only see a factor of 2 between the image file size and the memory used by the image data. This agrees with the observation that packed arrays of integers use 32 bits per element.


To confirm this, a ConstantArray containing values of 231−1 (the maximum signed 32-bit integer) is packed and has a ByteCount corresponding to 32 bits per element, whereas increasing the value by 1 (to a 33 bit number) prevents packing and the array now takes 512 bits per element to store.


bitinfo = Module[{arr = ConstantArray[#, {1000}]},
    {If[Developer`PackedArrayQ[arr], "Packed", "Not packed"],
     ToString[Floor[ByteCount[arr], 1000]/125] <> " bits"}] &;

bitinfo[2^31 - 1]

(*  {Packed, 32 bits}  *)

bitinfo[2^31]
(*  {Not packed, 512 bits}  *)

Note that small integers still use a 32 bit representation, even though they could be stored using fewer bits.


bitinfo[1]
(*  {Packed, 32 bits}  *)

To understand what is happening with images and tiff files, I wrote this:



With[{f = "testFile.tif", data = ConstantArray[0, {1000, 1000}], 
  types = {"Bit", "Byte", "Bit16", "Real32", "Real"}}, 
 Column[{"Bits per pixel for different image types:\n", 
   TableForm[
    Table[Round[
        8 {ByteCount@#, Export[f, #]; FileByteCount@f, 
           ByteCount@Import[f], ByteCount@Import[f, "Data"]}/10^6] &@
      Image[data, type], {type, types}], 
    TableHeadings -> {types, {"Initial\nImage", "Exported\nfile", 
       "Imported\nImage", "Imported\ndata"}}]}]]


enter image description here


The data shows that the image created by Image[data, type] uses a bit depth corresponding to the specified type (with the exception of "Bit" images which use 8 bits per pixel). The exported tiff file uses either 8 bits or 16 bits per pixel (I think these are the only bit depths supported by Export for tiff files). The imported Image object has the same bit depth as the file, and as expected the packed array of image data uses a bit depth of 32.


The interesting thing is that Mathematica can use 8-bit or 16-bit integers for Image objects, but packed arrays of integers are always 32-bit. So it's normal that the same data requires more memory as an array than as an Image.


The mystery is why you are getting a factor of 4 between a 16-bit image and the corresponding array. Perhaps the bit depth of packed arrays has increased to 64 bits in version 9? Please consider running the code in this answer and reporting what you find.


Comments

Post a Comment

Popular posts from this blog

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more

mathematical optimization - Minimizing using indices, error: Part::pkspec1: The expression cannot be used as a part specification

I want to use Minimize where the variables to minimize are indices pointing into an array. Here a MWE that hopefully shows what my problem is. vars = u@# & /@ Range[3]; cons = Flatten@ { Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; Minimize[{Total@((vec1[[#]] - vec2[[u[#]]])^2 & /@ Range[1, 3]), cons}, vars, Integers] The error I get: Part::pkspec1: The expression u[1] cannot be used as a part specification. >> Answer Ok, it seems that one can get around Mathematica trying to evaluate vec2[[u[1]]] too early by using the function Indexed[vec2,u[1]] . The working MWE would then look like the following: vars = u@# & /@ Range[3]; cons = Flatten@{ Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; NMinimize[ {Total@((vec1[[#]] - Indexed[vec2, u[#]])^2 & /@ R...
Post a Comment
Read more

What is and isn't a valid variable specification for Manipulate?

I have an expression whose terms have arguments (representing subscripts), like this: myExpr = A[0] + V[1,T] I would like to put it inside a Manipulate to see its value as I move around the parameters. (The goal is eventually to plot it wrt one of the variables inside.) However, Mathematica complains when I set V[1,T] as a manipulated variable: Manipulate[Evaluate[myExpr], {A[0], 0, 1}, {V[1, T], 0, 1}] (*Manipulate::vsform: Manipulate argument {V[1,T],0,1} does not have the correct form for a variable specification. >> *) As a workaround, if I get rid of the symbol T inside the argument, it works fine: Manipulate[ Evaluate[myExpr /. T -> 15], {A[0], 0, 1}, {V[1, 15], 0, 1}] Why this behavior? Can anyone point me to the documentation that says what counts as a valid variable? And is there a way to get Manpiulate to accept an expression with a symbolic argument as a variable? Investigations I've done so far: I tried using variableQ from this answer , but it says V[1...
Post a Comment
Read more