Skip to main content

Explaining error when using Part inside Module in Dataset


I have a vague recollection of seen an explanation for this, but I can not find it, so it may be a false memory. Will delete if duplicated.


Let ds be a simple Dataset



ds = Dataset@Table[
   <|"index" -> i, "data" -> RandomReal[1, 4]|>
   , {i, 3}]

This does NOT work (error message: k is not a valid part specification, Tag: dataset)


ds[All, Module[{k = 2, l}, l = #data[[k]]; <|#, "part2" -> l|>] &]


Mathematica graphics




But these very similar expression do work fine


ds[All, Module[{k = 1, l}, <|#, "part2" -> #data[[k + 1]]|>] &] (* Example 1 *)

ds[All, <|#, "part2" -> Module[{k = 2, l}, l = #data[[k]]]|> &] (* Example 2 *)

ds[All, Module[{k = 1, l},l = Extract[#data, {k + 1}]; <|#, "part2" -> l|>] &] (* Example 3 *)


Mathematica graphics




Using Block instead of Module gives the same result.


Where does this error come from?


Using Mathematica 11.0.1 on Windows 7 64 bit.



Answer



Cause


The problem is caused by a type-inferencing failure. When presented with a complex expression, the type inferencer will sometimes give up and tag the expression as being of type UnknownType. But at other times, the inferencer will fail outright. Expressions of the form <|#, ... |> or Extract[...] are identified as UnknownType. In constrast, expressions like expr[[k]] fall into the "failure" category when k is a symbol.


The expression that fails contains a "naked" reference to #data[[k]]. In working examples #1 and #2, the problematic part reference is contained within an outer expression <|#, ... |> -- so the type inferencer gives up (yielding UnknownType) before the part reference can fail. Working example #3 omits the Part reference altogether and uses Extract instead (which also yields UnknownType).


Work-around


The usual work-around for Dataset type system failures applies... dodge the type system by using Query on the raw data and wrapping Dataset around the result for visualization:


ds //

  Normal //
  Query[All, Module[{k=2, l}, l=#data[[k]]; <|#, "part2"->l|>]&] //
  Dataset

dataset screenshot


Analysis


current as of Mathematica v11.1.0


As part of the operation of a Dataset query, the system first performs some type checks to do some "sanity checks". In the case at hand, these checks fail outright:


ds[All, Module[{k = 2, l}, l = #data[[k]]; <|#, "part2" -> l|>] &]
(* Failure[...] *)


We can reduce this to a smaller example:


ds[All, Module[{k = 1}, #data[[k]]] &]
(* Failure[...] *)

Or even:


Dataset[{0}][Module[{k = 1}, #[[k]]] &]
(* Failure[...] *)

The cause is a failure of the type-inferencing function TypeApply:



Needs["TypeSystem`"]

TypeApply[{0}[[k]]&, {}]
(* FailureType[{Part, "Spec"}, <|"Type" -> Tuple[{Atom[Integer]}], "Part" -> k|>] *)

The full sequence of events looks like this:


stack excerpt


The component that is letting us down is TypePart. It can handle indices that are scalars, vectors or types:


TypePart[Tuple[{Atom[Integer]}], 1]
(* Atom[Integer] *)


TypePart[Tuple[{Atom[Integer]}], {1, 1}]
(* Tuple[{Atom[Integer], Atom[Integer]}] *)

TypePart[Tuple[{Atom[Integer]}],Atom[Integer]]
(* AnyType *)

But it fails outright when presented with a symbol:


TypePart[Tuple[{Atom[Integer]}], k]
(* FailureType[...] *)


I think it is arguable that this is a bug... a symbolic index should probably result in AnyType under the assumption that it will evaluate to a valid index at run time.


The working examples (1, 2, and 3) all "hide" the problematic part reference from the type inferencer by surrounding it by a troublesome, but not failing, outer expression:


TypeApply[<|#, "a" -> #a[[k]]|> &, DeduceType /@ {<|"a" -> {1, 2}|>}]
(* UnknownType *)

TypeApply[Extract[#a, {k + 1}] &, DeduceType /@ {<|"a" -> {1, 2}|>}]
(* UnknownType *)

The type inferencer is still confused, returning UnknownType, but at least it does not fail outright. The vague type information has no bearing upon the query evaluation process so we get the expected results. (UnknownType has been known to cause Dataset rendering issues from time-to-time.)



The Dataset type system tends to be (overly?) conservative when type-checking, but Query does no type-checking at all. So Query remains a viable work-around for these kinds of errors.


Comments

Post a Comment

Popular posts from this blog

mathematical optimization - Minimizing using indices, error: Part::pkspec1: The expression cannot be used as a part specification

I want to use Minimize where the variables to minimize are indices pointing into an array. Here a MWE that hopefully shows what my problem is. vars = u@# & /@ Range[3]; cons = Flatten@ { Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; Minimize[{Total@((vec1[[#]] - vec2[[u[#]]])^2 & /@ Range[1, 3]), cons}, vars, Integers] The error I get: Part::pkspec1: The expression u[1] cannot be used as a part specification. >> Answer Ok, it seems that one can get around Mathematica trying to evaluate vec2[[u[1]]] too early by using the function Indexed[vec2,u[1]] . The working MWE would then look like the following: vars = u@# & /@ Range[3]; cons = Flatten@{ Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; NMinimize[ {Total@((vec1[[#]] - Indexed[vec2, u[#]])^2 & /@ R...
Post a Comment
Read more

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more

What is and isn't a valid variable specification for Manipulate?

I have an expression whose terms have arguments (representing subscripts), like this: myExpr = A[0] + V[1,T] I would like to put it inside a Manipulate to see its value as I move around the parameters. (The goal is eventually to plot it wrt one of the variables inside.) However, Mathematica complains when I set V[1,T] as a manipulated variable: Manipulate[Evaluate[myExpr], {A[0], 0, 1}, {V[1, T], 0, 1}] (*Manipulate::vsform: Manipulate argument {V[1,T],0,1} does not have the correct form for a variable specification. >> *) As a workaround, if I get rid of the symbol T inside the argument, it works fine: Manipulate[ Evaluate[myExpr /. T -> 15], {A[0], 0, 1}, {V[1, 15], 0, 1}] Why this behavior? Can anyone point me to the documentation that says what counts as a valid variable? And is there a way to get Manpiulate to accept an expression with a symbolic argument as a variable? Investigations I've done so far: I tried using variableQ from this answer , but it says V[1...
Post a Comment
Read more