Skip to main content

Possible bug involving Dataset/Query and RightComposition


Bug introduced in 10.0.0 and fixed in 10.0.1



Short Version


Why are the results of the following expressions different?


Dataset[{1, 2}][f /* g, z]
Dataset[<|x->1, y->2|>][Values /* f /* g, z]

(*
g[f[{z[1],z[2]}]]

g[f[z[{1,2}]]]
*)

Long Version


Consider the following dataset queries:


Dataset[{1, 2}][f , z]
Dataset[{1, 2}][f /* g, z]
Dataset[{1, 2}][f /* g /* h, z]

(*

f[{z[1],z[2]}]
g[f[{z[1],z[2]}]]
h[g[f[{z[1],z[2]}]]]
*)

The results are as expected, with the ascending operators f, z, etc. being applied from the lowest level outward. But now consider these queries:


Dataset[<|x->1, y->2|>][Values /* f, z]
Dataset[<|x->1, y->2|>][Values /* f /* g, z]
Dataset[<|x->1, y->2|>][Values /* f /* g /*h, z]


(*
f[{z[1],z[2]}]
g[f[z[{1,2}]]]
h[g[z[f[{1,2}]]]]
*)

This time, Values is used to ignore the keys in the assocation. The level 1 query operator is a composition of a descending operator with one or more ascending operators. The documentation has this to say:



When one or more descending operators are composed with one or more ascending operators (e.g. desc /* asc), the descending part will be applied, then subsequent operators will be applied to deeper levels, and lastly the ascending part will be applied to the result.




Yet note how in the second query the level two ascending operator z is no longer applied to the level two elements. Worse still, in the third query the level 2 ascending operator z is being applied after the level 1 ascending operator f.


The operator reordering is most apparent in the "compiled" forms of the queries:


Dataset`CompileQuery[Values /* f, z]
Dataset`CompileQuery[Values /* f /* g, z]
Dataset`CompileQuery[Values /* f /* g /*h, z]

(*
Dataset`WithOverrides@*GeneralUtilities`Checked[Values/*Map[z]/*f,Identity]
Dataset`WithOverrides@*GeneralUtilities`Checked[Values/*f/*Map[z]/*g,Identity]
Dataset`WithOverrides@*GeneralUtilities`Checked[Values/*f/*g/*Map[z]/*h,Identity]

*)

Is this a bug, or a gap in my understanding?


Update


In a comment below, @TaliesinBeynon confirms that this is a bug that will be fixed in v10.0.1.



Answer



Workaround


Even though I cannot explain the behaviour, I note that the level one ascending operators will remain right-composed by means of a level one subquery or by a subsequent query:


Dataset[<|x->1, y->2|>][Values /* Query[f /* g /*h], z]
Dataset[<|x->1, y->2|>][Values, z][f /* g /*h]


(*
h[g[f[{z[1],z[2]}]]]
h[g[f[{z[1],z[2]}]]]
*)

Comments

Post a Comment

Popular posts from this blog

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...
Post a Comment
Read more

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...
Post a Comment
Read more

mathematical optimization - Minimizing using indices, error: Part::pkspec1: The expression cannot be used as a part specification

I want to use Minimize where the variables to minimize are indices pointing into an array. Here a MWE that hopefully shows what my problem is. vars = u@# & /@ Range[3]; cons = Flatten@ { Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; Minimize[{Total@((vec1[[#]] - vec2[[u[#]]])^2 & /@ Range[1, 3]), cons}, vars, Integers] The error I get: Part::pkspec1: The expression u[1] cannot be used as a part specification. >> Answer Ok, it seems that one can get around Mathematica trying to evaluate vec2[[u[1]]] too early by using the function Indexed[vec2,u[1]] . The working MWE would then look like the following: vars = u@# & /@ Range[3]; cons = Flatten@{ Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; NMinimize[ {Total@((vec1[[#]] - Indexed[vec2, u[#]])^2 & /@ R...
Post a Comment
Read more