make debian packages of dependencies

[warner]

To release tahoe-0.7.0, we need debian packages of all runtime dependencies that aren't already in debian, so that the tahoe .debs will actually be usable.

That means:

• zfec-1.3.4
• pycryptopp-0.2.10
• foolscap-0.2.3

The zfec package should really depend upon argparse (/usr/bin/zfec requires it), so that means a debian package for argparse too.

Post-0.7.0, if we implement #47 and depend upon pyutil, then we'll need to make debian packages of pyutil-1.3.12 too.

Foolscap comes with its own debian packaging rules: just do 'make debian-feisty' or whatever.

Zooko's packages use setuptools, and I've been using the 'stdeb' utility to turn them into debian packages. It's not trivial. The first step is to install stdeb (perhaps from ​http://stdeb.python-hosting.com/), then have it generate a .orig.tar.gz, .dsc, and most importantly the .diff.gz debian files. The next step is to unpack the tarball and apply the patch, then run 'debuild -uc -us' or './debian/rules binary' or dpkg-buildpackage (or some other utility that I'm not familiar with) and see if it works. Note that it's best if a copy of the .orig.tar.gz is present in the parent of the working directory, because then debuild has something against which it can generate a new .diff file. Also keep a copy of the stdeb-produced .diff.gz somewhere else, since debuild will probably overwrite the one in the parent directory.

The resulting package must also be inspected to see if it has all the desired files: usually documentation, examples, and README files are missing and/or misplaced. Most importantly, the debian/control file needs to be updated to list all the debian dependencies that the package requires: it can usually figure out C/.so deps by itself, but it can't usually figure out python dependencies. Build-time dependencies also go into debian/control . Edit debian/rules and debian/control until things work the way they ought to, then run 'debuild'/etc again to regenerate the .diff.gz . Then copy the .orig.tar.gz and .diff.gz over to a new directory, unpack/patch/build to make sure it all works. Then the source .orig.tar.gz and .diff.gz get copied to buildslave@hanford:tahoe-debs/dists/DIST/main/source/ , and the .deb file gets copied to main/binary-i386/ .

This needs to be done for each debian/ubuntu release we support: edgy, feisty, gusty, etch, and sid. It would probably be nice to make amd64 versions as well, but we haven't ever done this before and it would require changes to our repository config to actually make them reachable.

The same .diff.gz can probably be used for multiple releases, although several minimum versions probably get recorded in the debian/control files, so if you choose this approach, you might want to run stdeb on our oldest supported platform (which would be edgy) rather than on the newest (sid). Either 1: copy the .orig.tar.gz and .diff.gz to a new host, run debuild, then upload everything to hanford, or 2: copy (on hanford) the .orig.tar.gz and .diff.gz (and .dsc) into DIST/main/source/ for all dists, then go to the debian build host and do 'apt-get update && apt-get source PACKAGENAME', which will grab the source from hanford and unpack it. Doing 'apt-get build-dep PACKAGENAME' will look at the build-time dependencies and install them, which can save a lot of time. If you choose this path, you'll only have to upload the .deb file to hanford.

Once you've added something to tahoe-debs/ , run 'make' in that directory and it will update the index and push a copy out to org.

Once something has been packaged, it is frequently sufficient to apply the old .diff.gz to the new release tarball, add an entry to debian/changelog, modify the version number that stdeb unfortunately embeds in debian/rules, then run debuild.

Debian build hosts:

• edgy: slave-edgy.allmydata.com (inside the firewall)
• feisty: tahoeslave-fesity (inside the firewall)
• gutsy: tahoecs2 (in colo)
• etch: slave-etch (inside the firewall)
• sid: fluxx (inside the firewall)

For reference, we use slave3.allmydata.com (in colo) as a dapper buildslave. We test tahoe on dapper but don't produce debian package for it (although I do make foolscap packages for dapper).

[warner]

ok, foolscap-0.2.3 debs have been created for i386 on all releases

[warner]

I started to work on the pycrypto debs, but the new pycrypto requires darcsver and setuptools_darcs and stuff, and that got to be too much work, so I think I'll leave it for zooko. I suspect zfec has the same requirements.

[zooko]

"To release tahoe-0.7.0, we need debian packages of all runtime dependencies that aren't already in debian, so that the tahoe .debs will actually be usable. ... This needs to be done for each debian/ubuntu release we support: edgy, feisty, gusty, etch, and sid."

Okay, I've just spent some time learning how to to this -- thank you, Brian, for the extensive notes -- and it seems like packaging things for Debian is a lot of work.

Could we de-prioritize this for v0.7.0?

My general strategy for Tahoe until now has been not to offer binaries, because that's more work for us, and I don't mind if Tahoe v0.7.0 is available only to that subset of people who are willing to build from source, and I would like to spend our development effort on improving the build-from-source process. (Which still needs a lot of work, as we speak.)

My intent had been to make Tahoe v0.7.0 more easily installable from source (and we've succeeded -- see the new ​install.html), and then to start offering binary packages in Tahoe v0.7.1.

When it is time for us to spend the development resources to support binary distributions, then my priorities for platforms to support would be: 1. Whatever Zandr wants to deploy Tahoe on (I guess that is Gutsy), 2. Windows, which would potentially help Mike, Dmitry, Greg Hazel, Fabrice (?), Peter, Richard Samantha, and lots of other people, 3. whatever platform Richard Samantha has in his lab. 4. More Debian-likes not already supported for Zandr as number 1 above, 5. Mac OS X (note that allmydata.com receives frequent requests for a Mac client).

So I accept responsibility for increasing the work-load of making Debian packages by my choice of how to split up Python packages. Part of my justification for doing that is that it decreases the work-load for development and for making Python packages, which is the kind of trade-off that led me to focus on source packaging this time around.

So I'm willing to spend my time and effort to support Debian, which is a platform that I love and which I consider very important, but I would like to prioritize other things (​The Roadmap) over this issue for v0.7.0.

By the way, both darcsver and setuptools_darcs are build requirements, not install requirements, so presumably we don't need to produce .deb's for them. Hopefully that will make things easier.

[warner]

Could we de-prioritize this for v0.7.0?

I'm willing to accept that (although I can't say I'm happy about the regression). The consequences are:

• production nodes (testnet, perfnet, loadnet, i.e. tahoebs1-5 and tahoecs2) will be unable to run from debian packages, and must therefore run from source. I'm unwilling to have production machines using code installed via easy_install, and I'm reluctant to have them use code from /usr/local : these machines are a test case for how Zandr would deploy servers, so if we're not going to run them from debian packages, the remaining option which is the easiest to keep straight and to clean up after is running from a local source tree.

• we should stop updating our APT repository with new Tahoe .debs, since they are unusable, and older ones (before these new dependencies went in) are more usable than the new ones. In fact we should figure out what the last known usable version is, and remove any .debs produced after that date. Of course we should keep the buildbot .deb builder running.. we can just turn off the automatic mirror-to-org script. We should also update the .deb making code in our tree to express the new dependencies, so that anyone who *does* try to use them gets a clear install error rather than a cryptic runtime error.

Both of those are pretty easy to manage. So, ok, 0.7.0 will not have functional .debs, but the first thing we do for 0.7.1 should be to make these work again.

[warner]

oh, and I concur that darcsver and setuptools_darcs do not require debian packages, as long as they are buildable in the Desert Island mode: the debian/rules code that build the tahoe .deb will need to build these as well, to figure out the version number that should be put into the debian/changelog (and thus determine the .deb's version number).

[warner]

Hm, you might have lucked out: according to our calc-deps.py, tahoe only currently requires zfec-1.0.3 and pycryptopp-0.2.8 (even though we ship newer versions of both as tarballs in misc/dependencies/). We already have zfec-1.1 .debs for all platforms, and we have a pycryptopp-0.2.8 for feisty (which means that I've already done the necessary debianization work for that version). The current versions of zfec and pycryptopp are harder to build because of the new dependencies, but if we can get away with using the older versions for tahoe-0.7.0, then we can yet manage to have 0.7.0 be debian installable.

I'll look at the pycryptopp-0.2.8 feisty .deb and see how much work it will take to build it on the other platforms.

[warner]

ok, we've now got pycryptopp-0.2.8 debs on all platforms. I think this will be enough to allow tahoe .debs to be installed and run.

[zooko]

As long as we can run production servers against zfec-1.1 and pycryptopp-0.2.8, then we are currently okay on this issue, so I'm moving this to Milestone 0.10.0.

[warner]

This has become critical again, since tahoe trunk now depends upon a newer version of pycryptopp than we have .debs for. We'll need those .debs to update our production machines to code beyond 1.0.0

[zooko]

Okay! I can produce a python-pycryptopp-0.5.2-1_amd64.deb just by running ./setup.py sdist_dsc && cd deb_dist/pycryptopp-0.5.2 && dpkg-buildpackage -rfakeroot -uc -us.

I've figured out how to configure pycryptopp so that the README.txt, COPYING.GPL and COPYING.TGPPL.html go into the right place (/usr/share/docs/python-pycryptopp.

Next, I want to figure out how to make the resulting .deb automatically depend on python-setuptools based on the fact that the setup.py specifies that pycryptopp install_requires setuptools:

​http://allmydata.org/trac/pycryptopp/browser/setup.py

This is non-trivial, because setuptools and debian don't use exactly the same namespace -- i.e. I need to write either a heuristic ("foo in setuptools world probably means python-foo in debian world") or a map. Hm... :-)

Well, okay, it might be best to just use the feature which is explicitly for this purpose in the stdeb config file... We'll see... :-)

[zooko]

Okay, we have new .deb's for zfec, argparse, pyutil, and pycryptopp, and we have an easy way to make more of them when desired, so we're closing this ticket.

See also follow-on tickets #422 (stdeb: run from buildslaves), #423 (stdeb: use stdeb on tahoe itself), and #424 (stdeb: push to upstream).