functions - Strange behavior with Cases and Position with SetDelayed

I feel like I must be missing something simple and obvious here, but this has me scratching my head.

This works as expected:

list = {f[a], f[b]};
Cases[list, f[x_] :> x] -> Position[list, f[_]]

(* {a, b} -> {{1}, {2}} *)

However, this does not:

fun[list_] := Cases[list, f[x_] :> x] -> Position[list, f[_]];
fun[list]


(* {x, x} -> {{1}, {2}} *)

Is this a bug, or have I just not had enough coffee today?

Answer

What's happening

This is not simple by any means. You have encountered another instance of a general situation with lexical scope leaks / emulation / over-protection by symbol renaming. The case at hand is pretty similar to the one discussed here, so you can read the detailed explanation of this behavior in my answer there.

Roughly speaking, outer lexical scoping constructs (RuleDelayed in the linked dicsussion, and its analog for implicit global rule application here), try to protect the inner bindings from destructive changes, but mis-interpret their pieces and instead destroy yet inner bindings in the process. We have to fool that mechanism somehow, to avoid that.

Workarounds

The "StrictLexicalScoping" system option

Thanks to the hard work of Daniel Lichtblau, we now have a system option named "StrictLexicalScoping", which, when set to True, fixes many such cases, including the one at hand. You have to execute this:

SetSystemOptions["StrictLexicalScoping" -> True]

before you enter the definition of your function, and then it will work as intended.

Fooling the protection mechanism explicitly

In your case, here is one possible such work-around that is reasonably clean:

fun[list_] := 
   With[{rule = Rule}, 
      rule[Cases[list, f[x_] :> x], Position[list, f[_]]]
   ];

There are many more variations of it. What really matters is that SetDelayed and then the internal rule application engine (internal analog of RuleDelayed for global rules) don't see external Rule during the rule application.

This is surely not something that would first come to mind, though :)

Additional references

Here are a few additional links relevant to this discussion