upcoming build changes: moving to pip/tox/virtualenv

[Brian Warner]

Hey everybody.. just wanted to give y'all a heads-up on some
build/packaging changes that are about to land.

When we started Tahoe (over 9 years ago!), pip and virtualenv didn't
exist, so to make it easy to run tests and execute from a source tree,
we built our own system. Dependencies got installed into "support/", and
the bin/tahoe executable knew how to add the right directory into
sys.path . Back then, Tahoe sometimes spawned subprocesses, so we needed
to set both $PATH and $PYTHONPATH for their benefit. Running unit tests
required this kind of environment, so "setup.py test" knew how to run
"bin/tahoe" with a magic "@" argument that ran 'trial' in a
correctly-mangled subprocess environment. Eventually setuptools came
along, but it was mired in problems, and wasn't getting updated fast
enough for our needs, so we forked a copy (named "zetuptoolz") and
embedded the code into the tahoe source tree.

It was our own special snowflake of a build system, that's for sure. But
we've outgrown it, and by this point it's causing more trouble than it's
worth. Fortunately the python packaging world has raced ahead, and now
offers a suite of standard tools for us to use. In particular, we can
use "virtualenv" for creating a local execution environment, "pip" to
identify and install dependencies, and "tox" to run unit tests in a
consistent isolated environment.

So, once Daira lands the "1582.setuptools-delenda-est" branch, the
entire process for running tahoe's unit tests, starting from scratch,
without installing anything to the system or touching anything outside
the parent directory, will be:

 % git clone https://github.com/tahoe-lafs/tahoe-lafs.git
 % cd tahoe-lafs
 % tox

This makes some assumptions, which we're in the process of documenting:

* You need 'tox' installed first, which depends on 'virtualenv'.
* You'll be able to compile (or download pre-compiled wheels of) all
  dependencies that have C extensions. This includes pycryptopp, zfec,
  pyopenssl, cryptography, and some pieces of twisted.
  * Compiling those dependencies will probably require some C libraries
    and headers (and the python development headers) to be installed on
    your system. On a debian-like system, 'apt-get install libffi-dev
    libssl-dev python-dev'.
  * On windows and OS-X there may be pre-compiled wheels for these
    packages. We're trying to get the tahoe ones (pycryptopp/zfec) built
    and published to PyPI, so these platforms won't need a compiler.

To hack on Tahoe, the easiest approach will be to create a virtualenv
and install tahoe into it. Use --editable mode so you won't have to
re-install after making changes to the source tree.

 % cd tahoe-lafs
 % virtualenv tahoe-venv
 % source tahoe-venv/bin/activate
 (tahoe-venv) % pip install --editable .
 (tahoe-venv) % tahoe --version

If you just want to run the latest release, I can highly recommend the
"pipsi" tool[1], which installs python applications into a new
virtualenv and then symlinks the executables into your ~/.local/bin/
directory (which you can either add to your $PATH or symlink individual
targets into ~/bin):

 % pipsi install tahoe-lafs
 % ln -s ~/.local/bin/tahoe ~/bin/tahoe
 % tahoe --version

And if, for some reason, you want to install Tahoe (and its
dependencies) to /usr/local where everyone on your system can run it,
just do:

* sudo pip install tahoe-lafs

We have to update the buildbot (and our travis config) to match these
changes, so expect some redness for the next week or two.

These cleanups are long-overdue, and turned out to be a blocker for
making the next release (specifically because of #2728, a build failure
on OS-X), so we decided it was finally time to fix this stuff.

Let me know if you run into any problems.

cheers,
 -Brian


[1]: https://pypi.python.org/pypi/pipsi