There are somewhat similar questions on here, but I didn't find a satisfying answer for this case: I import a 1.7 GB binary data file using the following command:
BinaryReadList["/path/to/file", "Real64"]
It runs fine but the memory usage peaks at 18 GB during the import process, which is about 10 times the file size. Is there a more efficient way to do this? Why does Mathematica consume that much memory?
edit: It turned out, that almost all the memory is consumed by displaying the output. When I add a semicolon after the command, the consumed memory is close to the file size. This might not be a bug, but I still wonder why Mathematica behaves like this.
here is a test case:
- generate a file with:
BinaryWrite["test.dat", ConstantArray[99, 1*^8], "Integer8"];
- quit the kernel to reset MaxMemoryUsed[] and execute the following:
membefore = MemoryInUse[];
BinaryReadList["test.dat", "Real64"]
(MaxMemoryUsed[] - membefore)/(UnitConvert[FileSize["test.dat"], "Bytes"][[1]])
- don't forget to
DeleteFile["test.dat"];
this basically divides the MaxMemoryUsed by the FileSize and outputs the factor. You will notice a big difference depending on whether you add a semicolon after BinaryReadList["test.dat", "Real64"]
or not.
(I use Mathematica 12 on Ubuntu)
Comments
Post a Comment