Good ways to organize and document collections of mathematica notebooks?

I've now been using Mathematica for about a year, and am starting to get a somewhat unwieldy collection of notebooks. Sometimes I've figured out how to do a particular task in one of them, and want to use that again, but have to search through what I've done.

To tackle this, I've cobbled together a script that generates an html page for all the notebooks that I've checked into my source code repository (I happen to be using github to store my notebooks, latex, and other source code), and can put comments in that generated html file about what I did in each notebook, including any special techniques that I learned to accomplish the task. By way of example, I can list these with something like on this page.

From a local copy of my source repository I generate something similar, allowing a click on the various notebooks to view them with the CDF player.

I'm able to navigate my collection of notebooks well enough this way, but was curious how other people tackle the same organizational problem?

Answer

There are two answers to this question, and only one has anything to do with directory structure, and that only mildly.

The first answer is to structure the notebook itself using the style groupings provided. This will greatly enhance your ability to find something within any given notebook. For instance, as a materials physicist I use a model known as tight binding, and as part of my learning about how to apply it to a particular crystal structure, I have a notebook that I treat as a lab notebook. It has three main sections: Setup, Preliminary Testing, and Full Testing all at the highest level of the hierarchy. The least organized section is the Preliminary Testing, but even there I make ample use of Text cells to record my thoughts and observations. The other two sections make full use of the hierarchy to organize and group my examinations of the tight-binding method.

The second answer is packages. If you have some code that you find yourself referring to often, put in a package. Here I tend to use a rule of three, if I have to refer to or rewrite something 3 or more times, it needs to be in a package. It is here that a directory structure is helpful. In my case, my packages are currently organized into three directories: Quantum, Utilities, and Wien2k (a software package I've written interface scripts for) for a total of 16 packages.

I try to make each package somewhat complete in that they provide a full range of services. For example, the Wien2k package for loading their crystal structure files provides a full range of functions for accessing the structured information in those files, but sadly not saving them, as I haven't had the need to write it. But, some packages are just buckets of loosely related functions, and they're perfect that way.