Skip to main content

Best practices for using the Testing Framework


I am looking for best practices and recommended workflows for using the Testing Framework. For example, lets say I am developing a Mathematica package for other users, it is hosted on GitHub and I wish to make it easy for other developers to contribute to repository.


The project includes a suite of (unit) tests that I would like to run as often as necessary. I would like to run them within FrontEnd and conveniently inspect the results (timings, which tests failed and why, etc), save them to file and also further automate testing in the future. I can think of few possible workflows but none of them ticks all the boxes.




  • "Testing Notebook", Pro: It gives a clear overview of tests and looks nice. It can be run in FrontEnd and also programmatically with TestReport. Con: Contains even more metadata than the normal notebook (all those buttons) so it is less suitable to put under version control. Somehow the evaluation of tests is slower in testing notebook than in normal notebook.





  • ".wlt file", Pro: This is the documented way to use plain text files with test and run them programmatically from another notebook or command line. Con: I can't open .wlt files in FrontEnd!? So I have to open them in other text editor without nice features of FrontEnd (autocomplete, code coloring, etc).




  • ".wl file", Pro: I can easily edit it in FrontEnd as any other package. Con: This workflow is not documented. Should I put all tests in some specific Context?




So the question is, what is your recommended workflow for testing? While browsing other people's GitHub repositories I have seen some custom made approaches, but I would like rely on something that is documented and easy to explain to other developers who will contribute to the project.


I am also curious what is the purpose of BeginTestSection and EndTestSection in .wlt files?




Answer



I think @Jason B.'s answer is a good workflow. Here is an alternative that I have adopted over the years. Most of my code I write in *.m / *.mt / *.wl / *.wlt files using IntelliJ Idea with the Mathematica Plugin, so version control is straightforward.


My projects are written as paclets (see How to distribute Mathematica packages as paclets? for an introduction). Paclets allow you to specify resource files in the PacletInfo.m file. For every (major) function, I write a test-file .wlt or .mt and add it to the Paclet. Here is an example from my Multiplets paclet:


Paclet[
  Name -> "Multiplets",
  Version -> "0.1",
  MathematicaVersion -> "11.0+",
  Description -> "Defines multiplets and Young tableaux.",
  Thumbnail -> "multiplets.png",
  Creator -> "Johannes E. M. Mosig",

  Extensions -> {
    {"Kernel", Root -> ".", Context -> "Multiplets`"},
    {"Resource", Root -> "Testing", Resources -> {
        "MultipletDimension.mt",
        "MultipletQ.mt",
        "MultipletReduce.mt",
        "Tableau.mt",
        "TableauAppend.mt",
        "TableauClear.mt",
        "TableauDimension.mt",

        "TableauFirst.mt",
        "TableauFromMultiplet.mt",
        "TableauQ.mt",
        "TableauRest.mt",
        "TableauSimplify.mt",
        "TableauToMatrix.mt",
        "TableauToMultiplet.mt"
      }
    }
  }

]

In Mathematica, I can then load my paclet


<< Multiplets`

and check if any given function works as expected


TestReport@PacletResource["Multiplets", "MultipletReduce.mt"]

I can also open the test file within Mathematica, using


NotebookOpen@PacletResource["Multiplets", "MultipletReduce.mt"]


The front end opens *.mt and *.wlt files as plain text files, however, so if the notebook interface is important to you, then you may want to safe them as *.wl files instead.


In addition, one may add a test-script that just calls all other test scripts, for convenience.


Comments

Post a Comment

Popular posts from this blog

plotting - Plot 4D data with color as 4th dimension

I have a list of 4D data (x position, y position, amplitude, wavelength). I want to plot x, y, and amplitude on a 3D plot and have the color of the points correspond to the wavelength. I have seen many examples using functions to define color but my wavelength cannot be expressed by an analytic function. Is there a simple way to do this? Answer Here a another possible way to visualize 4D data: data = Flatten[Table[{x, y, x^2 + y^2, Sin[x - y]}, {x, -Pi, Pi,Pi/10}, {y,-Pi,Pi, Pi/10}], 1]; You can use the function Point along with VertexColors . Now the points are places using the first three elements and the color is determined by the fourth. In this case I used Hue, but you can use whatever you prefer. Graphics3D[ Point[data[[All, 1 ;; 3]], VertexColors -> Hue /@ data[[All, 4]]], Axes -> True, BoxRatios -> {1, 1, 1/GoldenRatio}]
Post a Comment
Read more

plotting - Filling between two spheres in SphericalPlot3D

Manipulate[ SphericalPlot3D[{1, 2 - n}, {θ, 0, Pi}, {ϕ, 0, 1.5 Pi}, Mesh -> None, PlotPoints -> 15, PlotRange -> {-2.2, 2.2}], {n, 0, 1}] I cant' seem to be able to make a filling between two spheres. I've already tried the obvious Filling -> {1 -> {2}} but Mathematica doesn't seem to like that option. Is there any easy way around this or ... Answer There is no built-in filling in SphericalPlot3D . One option is to use ParametricPlot3D to draw the surfaces between the two shells: Manipulate[ Show[SphericalPlot3D[{1, 2 - n}, {θ, 0, Pi}, {ϕ, 0, 1.5 Pi}, PlotPoints -> 15, PlotRange -> {-2.2, 2.2}], ParametricPlot3D[{ r {Sin[t] Cos[1.5 Pi], Sin[t] Sin[1.5 Pi], Cos[t]}, r {Sin[t] Cos[0 Pi], Sin[t] Sin[0 Pi], Cos[t]}}, {r, 1, 2 - n}, {t, 0, Pi}, PlotStyle -> Yellow, Mesh -> {2, 15}]], {n, 0, 1}]
Post a Comment
Read more

plotting - Mathematica: 3D plot based on combined 2D graphs

I have several sigmoidal fits to 3 different datasets, with mean fit predictions plus the 95% confidence limits (not symmetrical around the mean) and the actual data. I would now like to show these different 2D plots projected in 3D as in but then using proper perspective. In the link here they give some solutions to combine the plots using isometric perspective, but I would like to use proper 3 point perspective. Any thoughts? Also any way to show the mean points per time point for each series plus or minus the standard error on the mean would be cool too, either using points+vertical bars, or using spheres plus tubes. Below are some test data and the fit function I am using. Note that I am working on a logit(proportion) scale and that the final vertical scale is Log10(percentage). (* some test data *) data = Table[Null, {i, 4}]; data[[1]] = {{1, -5.8}, {2, -5.4}, {3, -0.8}, {4, -0.2}, {5, 4.6}, {1, -6.4}, {2, -5.6}, {3, -0.7}, {4, 0.04}, {5, 1.0}, {1, -6.8}, {2, -4.7}, {3, -1....
Post a Comment
Read more