remove the "build" step in the "edit, build, run" cycle

[zooko]

setup.py build -i ought to work

[warner]

Here's my plan:

• copy AES/SHA256 into src/allmydata, get rid of PyCrypto?: done
• make sure there are .debs for all our dependent libraries
  • this means zfec for all platforms, simplejson for the older ones that don't already have it. We know that Twisted, Nevow, and Foolscap already have .debs out there. We'll put copies of the .debs on the tahoe apt-repository for all platforms that don't already have them.
• Stop putting anything but tahoe code in the tahoe .deb: remove zfec and simplejson.

Once this is in place, it will be possible to install and use a tahoe .deb that does not contain anything other than tahoe code. The debian dependencies will be fulfilled by installing those other .debs .

• make sure that PyPI knows about all our dependent libraries
  • this means that 'easy_install zfec' and 'easy_install foolscap' should work. (this is already the case with foolscap).
  • Twisted doesn't work this way, I think it is too large and in too many pieces to be installed with easy_install. That's ok.
• change our setup.py to use setuptools, and declare a dependency on all our dependent libraries (except Twisted).

At this point, 'python setup.py install' should handle dependencies automatically. Our README can be changed to instruct users to make sure that python, setuptools, and Twisted are installed, then just type 'python setup.py install'. The 'make install' target should be a simple alias for this. I'd like to create a small script that adds the necessary --single-version-externally-managed and --prefix and PYTHONPATH=prefixything to make it possible to install tahoe code to somewhere other than /usr/lib (misc/install-to-prefix.py comes to mind), and then have the 'make install PREFIX=x' target be an alias for that.

In addition, the 'python setup.py build_ext -i' command will work: it will compile the crypto pieces to .so files in place. The 'make' target should be a simple alias for this.

• Next we should remove all non-tahoe source from our tree.
• Then we modify the bin/allmydata-tahoe script to add "src" to PYTHONPATH when it detects that we're running it from the source directory (at the moment it adds instdir/lib in this case).

With that, users who have all the necessary dependent libraries installed (perhaps because they ran 'easy_install zfec', etc) can run tahoe from source by just typing 'make; ./bin/tahoe COMMAND'.

• Next, we create a utility (misc/build-deps.py) that looks to see which of our dependent libraries are needed ('try: import zfec; except ImportError?: needed=True'), and for each one, run easy_install.py and tell it to put the results in ./support . The 'make build-deps' target should be an alias for this utility.
• Then we modify bin/allmydata-tahoe to add src/ to the beginning of PYTHONPATH and support/ to the end of PYTHONPATH.

With this, users who do not have the necessary dependent libraries can still run from source, by typing 'make; make build-deps; ./bin/tahoe COMMAND'.

• For convenience, we could put a copy of the source tarballs on the tahoe website, and change setup.py to add that as a possible download path: this would allow a tahoe tree to be built on a system that does not already have the dependent libraries installed even if the main zfec/simplejson/foolscap site were down. If we put a copy inside the tahoe source tree itself, this enables these sorts of builds without network access.
• Finally, we register tahoe on PyPI.

This last step would enable 'easy_install tahoe', again assuming that Twisted is already installed.

[zooko]

Excellent plan, Brian!

I'm making this ticket the successor to #15. See also the Packaging page.

This is the biggest component of the "packaging/build system" task which I hope will go into the v0.6 release.

[warner]

For reference, ​http://wiki.debian.org/DebianPythonFAQ suggests that --single-version-externally-managed is the Right Way to build debian packages out of setuptool-using python projects, and that the process is actually fairly easy.

[warner]

progress thus far:

• we have i386 zfec .debs for sid, edgy, and feisty. These packages are good enough to do 'import zfec', which is all we need for tahoe. They do not support the zfec/zunfec command-line tools (they do not even include them). Zandr is setting up an etch buildslave now (#24); once that's in place we can make a .deb for etch too.
• simplejson .debs already exist in feisty, edgy, etch, and sid.
• we have foolscap-0.1.4 .debs for sid, edgy, and feisty. This is enough for tahoe-0.5.0 . (foolscap-0.1.5 is the current release, but there are bugs in its debian packaging, and tahoe-0.5.0 doesn't need any of the new features, and foolscap-0.1.4 is wire-compatible with foolscap-0.1.5)

The next step will be to change the tahoe .deb generation to declare dependencies upon python-zfec, python-simplejson, and python-foolscap, and stop including zfec in the tahoe .deb .

[warner]

I've modified .deb generation. 'apt-get install allmydata-tahoe' now works, pulling in foolscap/zfec/simplejson as necessary.

The next step requires that 'easy_install zfec' works, which will require zfec-1.0.2 to be published (to fix the MANIFEST problems that caused ez_setup.py and src/fec.h to be left out of the source tarball, as well as to remove the setup.py-declared dependency on "pyutil").

Assigning to zooko for zfec-1.0.2, please assign back to me when the release is complete.

[zooko]

Okay, zfec-1.0.3 is easy_installable.

[warner]

we just implemented most of the rest of this today.

The pieces that remain:

• register the next release of tahoe with the cheeseshop
• add 'nevow' to the list of dependencies in setup.py
• identify the platforms on which setuptools is too old, preventing ez_setup from working
  • convince the author of simplejson to use our ez_setup patch to accomodate old setuptools
  • add back the simplejson source tarball to misc/dependencies/ , to enable desert-island building
• find a way to not build tahoe in the 'make build-deps' step, rather just build the dependencies
  • (this doesn't currently hurt anything, since we use the version from src/ before any others, but it's slow and confusing to look at)

[warner]

• added nevow
  • there was a problem on dapper, where the nevow that is available in the distribution is old (0.6.0) but works, but has no .egg-info, so our build-deps target didn't see it, and tried to build the new one (0.9.18), but that requires twisted-2.4.0 or newer, and the platform has twisted-2.2.0 . I hacked it to remove nevow from the dependencies if it sees nevow-0.6.0 available to import, which at least makes dapper work.
• stopped building tahoe in the 'build-deps' step, thanks to zooko's crazy/brilliant suggestion of using a dummy package named "tahoe-deps"

Now that the tahoe-specific things are done, I'm going to close out this ticket. I'll add a new one for registering tahoe with the cheeseshop.