packages - $Path hijacked by PacletManager?

Summary:

Since Mathematica version 6, the PacletManager seems to have engineered an escalating hijack of the $Path variable. Add-on packages need to have a PacletInfo.m file for their documentation to function correctly. However, use of this file also seems to trigger an override of the $Path variable's control of file loading. This breaks some of the functionality of Wolfram Workbench when using Mathematica 9 and 10.

Examples:

Suppose, as a minimal example, I have a Test directory in my $UserBaseDirectory/Applications directory that contains a PacletInfo.m file that reads

Paclet[
  Name -> "Test",
  Version -> "0.0.1"
]

as well as a file foo.txt containing the text "Applications directory".

Also, in my $HomeDirectory I have another Test directory containing just a text file foo.txt reading "Home directory".

Now, Mathematica 6 behaves as I would expect -- if I empty the $Path variable, Get can't find either foo.txt file, and if I put the $HomeDirectory in the $Path it will find the file in the $HomeDirectory (note: clearing the $Path variable will interfere with stuff like searching the documentation, so you probably want to run these in a fresh kernel and then restart):

$Path = {};    
Get["Test/foo.txt"]
(* During evaluation of In[2]:= Get::noopen: Cannot open Test/foo.txt. *)
(* $Failed *)

$Path = {$HomeDirectory};
Get["Test/foo.txt"]
(* "Home directory" *)

In Mathematica 7 or 8, there is a different behavior -- even with an empty $Path it will find foo.txt in the Applications directory, but with $HomeDirectory in the path it will find foo.txt in the $HomeDirectory, as before:

$Path = {};
Get["Test/foo.txt"]
(* "Applications directory" *)

$Path = {$HomeDirectory};
Get["Test/foo.txt"]
(* "Home directory" *)

In Mathematica 9 or 10, however, it always finds the text file in the Applications directory, even if only the $HomeDirectory is in the $Path:

$Path = {};
Get["Test/foo.txt"]
(* "Applications directory" *)

$Path = {$HomeDirectory};
Get["Test/foo.txt"]
(* "Applications directory" *)

With no PacletInfo.m file we get the expected (version 6) behavior in all versions. As a side note, there is no effect on the results if we change the name of the Test directory in the $UserBaseDirectory/Applications directory to something else; Get behaves as if foo.txt were in a directory with whatever name is listed in the PacletInfo.m file.

Discussion:

What seems to be happening is that, when a new kernel is started (or if we manually call PacletManager`RebuildPacletData[]), a list of paths is scanned for PacletInfo.m files, and the Name entries in those files are registered as directory search paths, corresponding to the actual directories in which the PacletInfo.m files are located. In recent versions these paths take precedence over the search paths in $Path.

The behavior in versions 9 and 10 interferes with the way Wolfram Workbench is supposed to work. When launching Mathematica from the Workbench, it puts the local development folder at the beginning of the $Path variable directory list, so that the development version of the package is loaded, rather than any installed version in the Applications directory. When the PacletInfo.m file present, however, the installed version is always loaded.

My questions:

1. 
   

   I'm curious if anyone has any insight as to whether the current behavior is considered desirable. Clearly a list of documentation directories has to be maintained, but is it necessary for the $Path to be overridden?

   
   


2. 
   

   Does anyone have a workaround to get Workbench to load the development version of a package? The obvious method is to either overwrite or delete the installed version in the Applications directory, but this is clumsy. More elegant would be to manually add the development directory to the beginning of the paclet search path, as well as to the $Path, but I haven't found a way of doing that yet.

   
   
   

Answer

From version 9 PacletInfo.m file requires "Kernel" extension with Context specification. Without it loading paclets using contexts doesn't work (see old answer).

Up to version 8 $Path has precedence over paclet search path no matter how package is loaded.

In versions 9.0 - 10.1 paclet search path has precedence over $Path no matter how package is loaded.

Since version 10.2 paclet search path is used and has precedence only when package is loaded using its context. When package is loaded using file path - paclet search path is not used at all - only $Path.

As noted in comment by Kuba, since version 9, when paclet is loaded using context, always newest version (determined by Version property from PacletInfo.m) from all available on paclet search path, is loaded, regardless of order of adding directories to paclet search path. Order of adding directories to paclet search path matters only when paclets in different directories have same version, then the one from directory added later has precedence.

Adding directory to the beginning of the paclet search path

One thing to note is that paclet search path is used only when getting files using a path with ordinary path separator. When getting packages using path with elements separated by context separator ` paclet search path is not used.

To add a directory to beginning of paclet search path in Mathematica versions 9.0 and 10.0 one can use PacletManager`PacletDirectoryAdd function.

To see how it works let's use slightly modified example from question i.e. instead of foo.txt let's use foo.m.

In Mathematica version 9 and 10 we get:

$Path = {};
Get["Test`foo`"]
(* Get::noopen: Cannot open Test`foo`. >> *)
(* $Failed *)
Get["Test/foo.m"]

(* "Applications directory" *)

(* Quit kernel *)
$Path = {$HomeDirectory};
Get["Test`foo`"]
(* "Home directory" *)
Get["Test/foo.m"]
(* "Applications directory" *)

(* Quit kernel *)

$Path = {};
PacletDirectoryAdd[$HomeDirectory];
Get["Test`foo`"]
(* Get::noopen: Cannot open Test`foo`. >> *)
(* $Failed *)
Get["Test/foo.m"]
(* "Home directory" *)

(* Quit kernel *)
$Path = {$HomeDirectory};

PacletDirectoryAdd[$HomeDirectory];
Get["Test`foo`"]
(* "Home directory" *)
Get["Test/foo.m"]
(* "Home directory" *)

So adding a directory to $Path and to paclet directories using PacletDirectoryAdd does the job and files will be always loaded from desired directory.

In Mathematica version 8 directories added with PacletDirectoryAdd are not put on the beginning of paclet search path, but it doesn't matter since in v8 ordinary $Path has precedence.

$Path = {};
Get["Test`foo`"]

(* Get::noopen: Cannot open Test`foo`. >> *)
(* $Failed *)
Get["Test/foo.m"]
(* "Applications directory" *)

(* Quit kernel *)
$Path = {$HomeDirectory};
Get["Test`foo`"]
(* "Home directory" *)
Get["Test/foo.m"]

(* "Home directory" *)

(* Quit kernel *)
$Path = {};
PacletDirectoryAdd[$HomeDirectory];
Get["Test`foo`"]
(* Get::noopen: Cannot open Test`foo`. >> *)
(* $Failed *)
Get["Test/foo.m"]
(* "Applications directory" *)


(* Quit kernel *)
$Path = {$HomeDirectory};
PacletDirectoryAdd[$HomeDirectory];
Get["Test`foo`"]
(* "Home directory" *)
Get["Test/foo.m"]
(* "Home directory" *)

Workaround for Workbench

To add elements of $Path, that are in Workbench workspace, to paclet search path one can create an init.m file with following contents:

If[$VersionNumber >= 9,
PacletManager`PacletDirectoryAdd @ Reverse @ Select[
    $Path,
        StringMatchQ[#, "/path/to/workspace/*"]&
    ]
]

Unfortunately I don't have a nice and clean method of executing this file every time it's needed.

A non-ideal solution is to start execution build command of each project with

Get["path/to/our/init.m"]