built in symbols - What's purpose of the new function BinarySerialize?

11.1 introduced a new function BinarySerialize, but I don't know what it can do better than the traditional method.Its behavior is very similar to Compress,though I cannot find any advantage of it.It even consumes more space than Compress,such as

BinarySerialize[Range[100]] // Normal // Length

805

Compress[Range[100]] // ToCharacterCode // Length

290

And ByteCount[BinarySerialize[Range[100]]] is also greater than ByteCount[Compress[Range[100]]].So what's purpose of this function? Anyone can provide a good example to use it?

Answer

Disclaimer: This answer is written from a user's point of view. For useful insider information on this topic see this discussion with Mathematica developers on Community Forums.

Introduction

Binary serialization is rewriting expressions as an array of bytes (list of integers from Range[0, 255]). Binary representation of expression takes less space than a textual one and also can be exported and imported faster than text.

How do Compress and BinarySerialize functions work?

Compress (with default options) always does three steps:

1. It performs binary serialization.



2. It deflates result using zlib.


3. It transforms deflated result to a Base64-encoded text string.

BinarySerialize performs only binary serialization and sometimes deflates the result using zlib. With default options it will decide itself if it wants to deflate or not. With an option PerformanceGoal -> "Speed" it will avoid deflation. With an option PerformanceGoal -> "Size" it will likely deflate. BinarySerialize returns a ByteArray object. ByteArray is something like a packed array of 8-bit integers. However FullForm of ByteArray is visualized as Base64-encoded text string. This visualization can be somewhat misleading, because internally ByteArrays are stored and operated in binary form, not as text strings.

Binary serialization algorithms of Compress and BinarySerialize

Original serialization algorithm of Compress is described in this answer. That algorithm is not very optimized for size and produces larger-then-necessary output for many typical expressions. For example, it has no support for packed arrays of integers and rewrites such arrays as nested lists, which take a lot bytes.

BinarySerialize uses a more size-optimized binary serialization algorithm compared to what Compress (with default options) does. This algorithm supports packed arrays of integers, has optimizations for integers of different size (8,16,32 bit), stores big integers in binary form (not as text strings), and has other optimizations.

Applications of BinarySerialize

Using BinarySerialize we can write our own Compress-like functions with better compression. For example we can write myCompress function which does the same three steps as original Compress, but uses BinarySerialize for the serialization step:

myCompress[expr_]:=Module[

    {compressedBinaryData},
    compressedBinaryData = BinarySerialize[expr, PerformanceGoal->"Size"];
    Developer`EncodeBase64[compressedBinaryData]
    ];

myUncompress[string_]:=Module[
    {binaryData},
    binaryData = Developer`DecodeBase64ToByteArray[string];
    BinaryDeserialize[binaryData]
    ];

Even for simple integer list we can see size reduction.

Compress[Range[100]] // StringLength
(* 290 *)

myCompress[Range[100]] // StringLength
(* 244 *)

myUncompress[myCompress[Range[100]]] === Range[100]
(* True *)

If we take an expression with large number of small integers we get much more noticeable improvement:

bitmap = Rasterize[Plot[x, {x, 0, 1}]];

StringLength[Compress[bitmap]]
(*31246*)

StringLength[myCompress[bitmap]]
(*17820*)


myUncompress[myCompress[bitmap]] === bitmap
(* True *)

Conclusion

The example above shows that the result of a simple user-defined function myCompress based on a BinarySerialize can be almost twice more compact than the result of Compress.

Outlook

To decrease the output size even further one can use a compression algorithm with higher compression settings (in the second step) or use Ascii85-encoding instead of Base64 in the third step.

Appendix 1: Undocumented options of Compress

I have noticed that in Version 11.1 Compress has more undocumented options than in previous versions. Those options allows one to:

• 
  

  Disable both compression and Base64 encoding and return a binary serialized result as a string with unprintable characters:

  
  

  Compress[Range[100], Method -> {"Version" -> 4}]

  
  


• 
  

  Change binary serialization algorithm to a more efficient one, but not exactly to BinarySerialize.

  
  

  Compress[Range[100], Method -> {"Version" -> 6}] // StringLength

  
  

  (* 254 *)

  
  

There is also a "ByteArray" option shown in usage message ??Compress but it does not work in Version 11.1.

Note that this behavior is undocumented and may change in future versions.

Appendix 2: Compression option of BinarySerialize

Just for fun one can manually compress result of BinarySerialize[..., PerformanceGoal -> "Speed"] to get the same output as BinarySerialize[..., PerformanceGoal -> "Size"] produces. This can be done with the following code:

myBinarySerializeSize[expr_]:=Module[
    {binaryData, dataBytes, compressedBytes},
    binaryData = Normal[BinarySerialize[expr, PerformanceGoal->"Speed"]];
    dataBytes = Drop[binaryData, 2]; (*remove magic "7:"*)
    compressedBytes = Developer`RawCompress[dataBytes];
    ByteArray[Join[ToCharacterCode["7C:"], compressedBytes]]

    ]

We can check that it gives the same result as PerformanceGoal -> "Size" option

data = Range[100];
myBinarySerializeSize[data] === BinarySerialize[data, PerformanceGoal -> "Size"]

Appendix 3: zlib compression functions

Description of undocumented zlib compression/decompression functions Developer`RawCompress and Developer`RawUncompress can be found in this answer.

Appendix 4: Base64 encoding functions

Usage of Base64 encoding/decoding functions from the Developer` context can be explained using the following code:

binaryData = Range[0, 255];

Normal[
    Developer`DecodeBase64ToByteArray[
        Developer`EncodeBase64[binaryData]
        ]
    ] == binaryData

(* True *)