Subprojects

A subproject is an independent project that is in active development which the main project makes use of. By active development, I mean that it cannot be effectively worked with using pip install subproject (in which case it will simply have a concrete requirement entered in requirements.txt).

No to submodules or subtrees

Traditionally, this sort of thing is handled using submodules or subtrees. We've tried both and there are still a few submodules skulking about. In short, they are a pain and the setup method we're adopting in their place is simpler and much easier to work with.

Yes to Python setup

A subproject is designed to be used within other projects. Therefore we set it up using Python setuptools which will ultimately allow it to be installed using pip install subproject and immediately allow us to install it in our main project using a variety of methods

Method	When to use the method
From requirements.txt	This is the default and is why this method is nice and simple. Unless you are directly concerned with the subproject, you no longer need to worry about it
python setup.py develop	This is where you are working on both the subproject and on the main project. With this installation, the egg in the main project links to the actual files in the subproject so it tracks changes as you make them.
pip uninstall subproject ; pip install -r requirements.txt	This will upgrade the subproject to the latest version specified in the requirements.txt file. Unfortunately, there is a bug in pip such that, unless the version number of the subproject has changed (in setup.py), the new commit entry will be ignored. Therefore you have to first uninstall the relevant egg with eg pip uninstall docmeta before calling `pip install -r requirements.txt.

Setting up the subproject

Other than ensuring that the subproject has a setup.py, there is no special setup required for the subproject.

Setting up the main project requirements.txt

This line:

-e [email protected]:cccs-web/docmeta.git#egg=docmeta-master

declares an abstract dependency between the main project and the docmeta subproject. If this command is added to the requirements.txt and pip install -r requirements.txt is invoked then the docmeta module will be installed along with any of its dependencies that are not already satisfied.

This line should not generally be used in a requirements.txt because the requirements.txt aims to present a 'concrete' specification of the versions - it should look more like this:

-e [email protected]:cccs-web/docmeta.git@eb408f9e9f7f667c8fc488542b576a627651bb8a#egg=docmeta-master

It is now specifying a specific commit (or tag) making it a concrete requirement. This removes the risk of an incompatible subproject change accidentally breaking the main project. It also allows the 'release' of a new subproject commit to the main project by changing the main project requirements.txt file appropriately.