Skip to main content

documentation - How has Hash changed in 11.3?


There are already few topic related to Hash[_String]:


How does Hash calculate hash for strings?


Incorrect calculating Hash SHA256


But it looks like changes are more severe:


Hash[{}] returns the same in V11.2 and V11.3, but e.g. Hash[{}, "MD5"] does not.


enter image description here


And I don't see an explanation in documentation:


enter image description here


What is the complete list of changes? How to make old code compatible with those changes?




Answer



There is already discussion about String and ByteArray in the linked previous Q & A, so I'll comment a bit about general expressions.


This only concerns the named hash algorithms like "MD5" or "SHA" etc.


The single argument form Hash[expr], which is equivalent to Hash[expr, "Expression"] is completely separate and based on the internal representation of expr. It has not been changed for 11.3.


Using @Kuba's example of a very simple expression, in 11.3 we have the following new hash value


Hash[{}, "MD5"]

(* 68244457821771821570522625853545795031 *)

The previous hash value can still be obtained using



Developer`LegacyHash[{}, "MD5"]

(* 272934427398090264974473461931457450337 *)

Why the change? The previous scheme for converting expressions (to strings) for hashing had a number of severe problems.


For example, it did not take into account the contexts of symbols so it could happen that the same expression would hash to a different value because of a different $ContextPath, it had issues with evaluation leaks etc.


These have now been addressed, but the fixes mean the hash values would inevitably change. Developer`LegacyHash is provided for people who in some way depend on the old hash values.


The wording in the documentation isn't fully accurate, because ToString[FullForm[expr]] is not used literally.


What is actually true is that the input given to the hashing algorithm is based on the bytes of ToString[Unevaluated[FullForm[expr]]], where all symbols in expr are qualified with their full contexts.


Furthermore, a constant 32-byte sequence prefix is added for (non-String and non-ByteArray) expressions to avoid collisions -- this ensures that the number 2 and the string "2" do not end up having the same hash value. This is because ToString[Unevaluated[FullForm[2]]] is the same as the string "2" but 2 and "2" are different expressions.



Below is a mock-up example (not the actual implementation) that could be used to replicate the 11.3 hash value even on earlier versions. It uses the byteHash utility defined in my previous answer.


prefix = {209, 74, 9, 190, 254, 30, 81, 99, 147, 98, 22, 44, 107, 239, 77, 113, 
          23, 185, 9, 18, 189, 28, 97, 183, 43, 63, 221, 103, 61, 127, 201, 101};

byteHash[Join[prefix, ToCharacterCode["System`List[]"]], "MD5"]

(* 68244457821771821570522625853545795031 *)

While the documentation ideally should give some idea of what serialization is used for general expressions, I would not hold the expectation that it must go into any deep level of detail or provide sufficient information to actually write an alternative implementation. Besides, the serialization could conceivably change some day again.


I think the moral is, if people want full control, they should themselves create a sequence of bytes to give as input to the hashing method in whatever way they see as appropriate. Then, what a named algorithm like "MD5" or "SHA" must return is fully determined, and a result different from that would certainly be a bug.



Comments

Post a Comment

Popular posts from this blog

mathematical optimization - Minimizing using indices, error: Part::pkspec1: The expression cannot be used as a part specification

I want to use Minimize where the variables to minimize are indices pointing into an array. Here a MWE that hopefully shows what my problem is. vars = u@# & /@ Range[3]; cons = Flatten@ { Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; Minimize[{Total@((vec1[[#]] - vec2[[u[#]]])^2 & /@ Range[1, 3]), cons}, vars, Integers] The error I get: Part::pkspec1: The expression u[1] cannot be used as a part specification. >> Answer Ok, it seems that one can get around Mathematica trying to evaluate vec2[[u[1]]] too early by using the function Indexed[vec2,u[1]] . The working MWE would then look like the following: vars = u@# & /@ Range[3]; cons = Flatten@{ Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; NMinimize[ {Total@((vec1[[#]] - Indexed[vec2, u[#]])^2 & /@ R...
Post a Comment
Read more

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more

What is and isn't a valid variable specification for Manipulate?

I have an expression whose terms have arguments (representing subscripts), like this: myExpr = A[0] + V[1,T] I would like to put it inside a Manipulate to see its value as I move around the parameters. (The goal is eventually to plot it wrt one of the variables inside.) However, Mathematica complains when I set V[1,T] as a manipulated variable: Manipulate[Evaluate[myExpr], {A[0], 0, 1}, {V[1, T], 0, 1}] (*Manipulate::vsform: Manipulate argument {V[1,T],0,1} does not have the correct form for a variable specification. >> *) As a workaround, if I get rid of the symbol T inside the argument, it works fine: Manipulate[ Evaluate[myExpr /. T -> 15], {A[0], 0, 1}, {V[1, 15], 0, 1}] Why this behavior? Can anyone point me to the documentation that says what counts as a valid variable? And is there a way to get Manpiulate to accept an expression with a symbolic argument as a variable? Investigations I've done so far: I tried using variableQ from this answer , but it says V[1...
Post a Comment
Read more