Skip to main content

list manipulation - Emulating R data frame getters with UpValues


What's the best way to emulate R's data frames functionality? This includes the ability to select rows and columns in a 2-dimensional table by the string identifiers positioned typically in the first row or column, see this related question, (and thanks to Leonid Shifrin for cluing me to the term 'data frame.')



Wrapping a 2D array in the symbol DF enables leveraging upvalues to essentially overload Part for the getters, here defined separately for singleton and list string arguments; matching uses Position.


To select rows at level 1 of the array:


Part[DF[expr_], r_String, rest___] ^:= 
expr[[Position[expr, r][[1, 1]], rest]]

Part[DF[expr_], r_List /; And @@ StringQ /@ r, rest___] ^:=
expr[[Position[expr, #][[1, 1]] & /@ r, rest]]

Similarly to get columns at level 2 of the array:


Part[DF[expr_], All, c_String] ^:= 

Transpose[expr][[Position[Transpose@expr, c][[1, 1]]]]

Part[DF[expr_], All, c_List /; And @@ StringQ /@ c] ^:=
Transpose[expr][[Position[Transpose@expr, #][[1, 1]] & /@ c]]

These seem to work correctly (though I've not implemented preconditions checks that the header strings are at the correct level or that the array Dimension length is 2), eg given data:


data = {{"ID", "COL1", "COL2", "COL3"}, {"ROW1", 1, 2, 3}, {"ROW2", 4,
5, 6}, {"ROW3", 7, 8, 9}};

yields:



DF[data][[{"ID", "ROW1", "ROW3"}]]

(* {{"ID", "COL1", "COL2", "COL3"}, {"ROW1", 1, 2, 3}, {"ROW3", 7, 8, 9}}*)

and


DF[data][[All, {"ID", "COL2"}]]

(* {{"ID", "ROW1", "ROW2", "ROW3"}, {"COL2", 2, 5, 8}} *)

Note that this mechanism is somewhat more elegant than R, where the string arguments are wrapped in the 'c' function.



Several related questions:


First, these definitions raise similar errors - why?


UpSetDelayed::write: "Tag String in DF[expr_][[r_String,rest___]] is Protected."

Second - key question - how to define general getter passing row & column arguments to return a subarray, eg DF[data][[{"ID","ROW1"},{"ID","COL2"}]] but avoiding replicating the list and singleton combinations above? Is there an implementation of the general case that avoids the special cases?



  • Arguments need not include the "ID" row or column, composing the row getter followed by column getter, or vice versa, won't work directly.

  • Note the asymmetry and limitation between the methods to access rows and columns: the row accessor is more general since All can be passed as a special case, but also numeric parameters and Spans.


Finally, can the UpValues mechanism be used to emulate other data frames functionality? Would it be possible to use as setters? Also, Apparently in R data frames are used to query the data with conditionals.



EDIT


Note, given the comments on computational complexity and scalability, I would like to stress that the primary purpose of this approach is to deal with small to medium datasets imported from Excel, where string headers, unlike RDBMS, are not separated as metadata.


In many data analysis applications, for example biomedicine and healthcare, Excel-based workflows are the norm (for input data, intermediates and output). Moreover, query (and statistical) development time dominates execution time. So syntax and modular approaches to data frames outweigh computational complexity considerations.




Comments

Popular posts from this blog

mathematical optimization - Minimizing using indices, error: Part::pkspec1: The expression cannot be used as a part specification

I want to use Minimize where the variables to minimize are indices pointing into an array. Here a MWE that hopefully shows what my problem is. vars = u@# & /@ Range[3]; cons = Flatten@ { Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; Minimize[{Total@((vec1[[#]] - vec2[[u[#]]])^2 & /@ Range[1, 3]), cons}, vars, Integers] The error I get: Part::pkspec1: The expression u[1] cannot be used as a part specification. >> Answer Ok, it seems that one can get around Mathematica trying to evaluate vec2[[u[1]]] too early by using the function Indexed[vec2,u[1]] . The working MWE would then look like the following: vars = u@# & /@ Range[3]; cons = Flatten@{ Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; NMinimize[ {Total@((vec1[[#]] - Indexed[vec2, u[#]])^2 & /@ R...

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...

What is and isn't a valid variable specification for Manipulate?

I have an expression whose terms have arguments (representing subscripts), like this: myExpr = A[0] + V[1,T] I would like to put it inside a Manipulate to see its value as I move around the parameters. (The goal is eventually to plot it wrt one of the variables inside.) However, Mathematica complains when I set V[1,T] as a manipulated variable: Manipulate[Evaluate[myExpr], {A[0], 0, 1}, {V[1, T], 0, 1}] (*Manipulate::vsform: Manipulate argument {V[1,T],0,1} does not have the correct form for a variable specification. >> *) As a workaround, if I get rid of the symbol T inside the argument, it works fine: Manipulate[ Evaluate[myExpr /. T -> 15], {A[0], 0, 1}, {V[1, 15], 0, 1}] Why this behavior? Can anyone point me to the documentation that says what counts as a valid variable? And is there a way to get Manpiulate to accept an expression with a symbolic argument as a variable? Investigations I've done so far: I tried using variableQ from this answer , but it says V[1...