Skip to main content

import - Why does Mathematica consume so much memory when reading binary data?


There are somewhat similar questions on here, but I didn't find a satisfying answer for this case: I import a 1.7 GB binary data file using the following command:


BinaryReadList["/path/to/file", "Real64"]


It runs fine but the memory usage peaks at 18 GB during the import process, which is about 10 times the file size. Is there a more efficient way to do this? Why does Mathematica consume that much memory?


edit: It turned out, that almost all the memory is consumed by displaying the output. When I add a semicolon after the command, the consumed memory is close to the file size. This might not be a bug, but I still wonder why Mathematica behaves like this.


here is a test case:



  1. generate a file with:


BinaryWrite["test.dat", ConstantArray[99, 1*^8], "Integer8"];



  1. quit the kernel to reset MaxMemoryUsed[] and execute the following:


membefore = MemoryInUse[];
BinaryReadList["test.dat", "Real64"]
(MaxMemoryUsed[] - membefore)/(UnitConvert[FileSize["test.dat"], "Bytes"][[1]])


  1. don't forget to


DeleteFile["test.dat"];


this basically divides the MaxMemoryUsed by the FileSize and outputs the factor. You will notice a big difference depending on whether you add a semicolon after BinaryReadList["test.dat", "Real64"] or not.


(I use Mathematica 12 on Ubuntu)




Comments

Post a Comment