error - Using Merge[Identity] and Transpose on a Dataset

While experimenting with this answer, I ran across the following:

data = Dataset[{
    <|"a" -> 1, "b" -> "x"|>,
    <|"a" -> 2, "b" -> "y"|>}];


data[Transpose /* f, {"a", "b"}]

(* f[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>] *)

data[Merge[Identity] /* f, {"a", "b"}]

(* f[<|"a" -> {1, 2}, "b" -> {"x", "y"}|>] *)

But when I fill in f with a function:

(*This works*)
data[Transpose /* (Max[#a] + Total[#b] &), {"a", "b"}]

(*This fails*)
data[Merge[Identity] /* (Max[#a] + Total[#b] &), {"a", "b"}]

Replacing #a and #b with #[[1]] and #[[2]] seems to work.

1. Why does using Merge[Identity] fail?


2. Is Transpose[] a descending operator in a Dataset, and is this related to 1.?

Answer

I believe what you experience should not happen because when you use the alternative notation for named slots, it works as expected:

data = Dataset[{<|"a" -> 1, "b" -> "x"|>, <|"a" -> 2, "b" -> "y"|>}];
data[Merge[Identity] /* (Max[#["a"]] + Total[#["b"]] &)]
(* 2 + "x" + "y" *)

Additionally, you could split your original function into two parts and it still works:

data[Merge[Identity]];
%[(Max[#a] + Total[#b] &)]

(* 2 + "x" + "y" *)

Therefore, without further inspection through Trace or friends, I would say this is a bug or a feature I don't understand. Nevertheless, I cannot answer your first question why it specifically fails.

To your second question: Transpose should be an ascending operator, when I understood the documentation and the meaning correctly. Let me illustrate this. Assume we have to following test Dataset:

test = Dataset[RandomInteger[9, {2, 3}]]

Now, we do some test with the descending operator All. Descending means, first all data is selected and the following min/max function is applied to each element on the next level, which are the vectors {5,7,9} and {8,3,7}

test[All, {Min[#], Max[#]} &] //Normal
(* {{5, 9}, {3, 8}} *)

The output are the expected min/max values of each vector.

Transpose works differently. First, the min/max function is calculated on all elements and then the result is transpose. And this, although Transpose comes first:

test[Transpose, {Min[#], Max[#]} &] // Normal
(* {{5, 3}, {9, 8}} *)

If Transpose would be a descending operator, the output should be

test[Transpose][All, {Min[#], Max[#]} &] // Normal

(* {{5, 8}, {3, 7}, {7, 9}} *)

but this is not the case.