[tahoe-dev] v1.5.0 -- now with less apt-get! Re: v1.5.0 is feature-complete!

[Brian Warner]

Zooko Wilcox-O'Hearn wrote:

> In the meantime, if anybody else knows how to build an apt repository
> (possibly by using Ubuntu PPAs?), please let us know -- our buildbots
> are providing all the .deb packages that we need (I think) -- we just
> need an apt repository to serve them.

Unfortunately, I don't think this is quite true. The directory where
we're currently uploading debs (our our internal machine named
"hanford") has the following recent Tahoe debs (note that r4031 is the
current trunk, but for whatever reason the cygwin buildslave is offline
and that's preventing our buildmaster from making debs):

 buildslave at hanford:~/tahoe-debs$ find . -name
'*allmydata-tahoe_1.4.1-r4026*'
 ./dists/etch/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb
 ./dists/feisty/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb
 ./dists/edgy/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb
 ./dists/gutsy/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb
 ./dists/hardy/tahoe/binary-i386/allmydata-tahoe_1.4.1-r4026_all.deb
 ./dists/hardy/tahoe/binary-amd64/allmydata-tahoe_1.4.1-r4026_all.deb

But only the following suitable pycryptopp debs:

 buildslave at hanford:~/tahoe-debs$ find . -name '*pycryptopp_0.5.16*'
 ./dists/sid/main/binary-i386/python-pycryptopp_0.5.16-1_i386.deb
 ./dists/jaunty/main/binary-amd64/python-pycryptopp_0.5.16-1_amd64.deb
 ./dists/lenny/main/binary-i386/python-pycryptopp_0.5.16-1_i386.deb

(also note that we are not yet indexing the jaunty or lenny debs: this
is the task I'll try to finish next week).

These sets are entirely disjoint! There is no (debian/ubuntu release) +
(architecture) combination for which you can get both a correct tahoe
deb and a pycryptopp deb.

What releases should we target? Really, I'd like us to offer pycryptopp
debs and tahoe debs for the same platforms: e.g. the gutsy tahoe deb is
useless without a gutsy pycryptopp deb. But to prioritize just a few:
Jaunty is the most recent Ubuntu release, Hardy is the most recent
Ubuntu LTS (long-term-support) release, and Lenny is the most recent
Debian release, so those three are arguably the most important to
support. I'll generally prioritize i386 over amd64, noting that tahoe
itself is pure-python (and it's only our poor APT-repo-management setup
that causes an "_all.deb" to get placed in a binary-i386/ directory), so
amd64 users can install a tahoe deb from an i386 directory.

Note that in general, you can probably use an older deb on a newer
system, but not vice versa. Using a lenny (debian) deb on a jaunty
(ubuntu) system or vice versa are both pretty uncertain: there is not a
total ordering between debian releases and ubuntu releases.

The best recommendations we can currently make are:

 Jaunty/amd64: probably workable: use the hardy tahoe deb and the jaunty
               pycryptopp deb [requires APT-repo indexing to be done]
 Jaunty/i386: fail: we have a hardy tahoe deb, but no pycryptopp deb
              (the sid is too new, the lenny is debian-vs-ubuntu)
 Hardy/i386: fail: we have a hardy tahoe deb, but no pycryptopp deb
 Hardy/amd64: fail: we have a hardy tahoe deb, but no pycryptopp deb
 Lenny/i386: dubious: we have the right pycryptopp, but best tahoe
             deb we can offer is from etch [also requires APT-repo
             indexing to be done]
 Lenny/amd64: fail: we don't have good debs for either tahoe or
              pycryptopp

So, none of these platforms will work right now, and even once I finish
APT-repo indexing task, we'll only provide marginal support for two out
of the six targets (jaunty/amd64 and lenny/i386).

To support these platforms properly, we need to create debs for:

 tahoe: lenny/i386, lenny/amd64, jaunty/i386, jaunty/amd64
 pycryptopp: lenny/amd64, hardy/i386, hardy/amd64, jaunty/i386

Since hardy debs should be installable on jaunty (and since hardy is the
latest LTS), and since lenny users can probably install etch debs, it
would be enough to build:

 tahoe: lenny/amd64
 pycryptopp: lenny/amd64, hardy/i386, hardy/amd64

allmydata.com has a host to provide hardy/i386 (deharo1), but we don't
yet have jaunty or lenny boxes, or very many amd64 hosts. If Zooko
doesn't do it before I get back, I'll copy a pycryptopp tarball over to
deharo1 and manually build a deb and copy it into our APT repo. That
will at least give us an answer for our two most important debian-ish
platforms: hardy/i386 and jaunty/i386.

For the future, I've been experimenting with pbuilder/schroot on an EC2
instance to create multiple buildslaves (one per supported release: from
dapper on up to jaunty and maybe the latest "karmic" snapshot) at the
same time, but it's likely to be at least a month before I've got that
stuff working well enough to be useful. (I'm also hoping to implement
automated installation testing with the same framework). When that's in
place, if I can find the time, I hope to clean up our debian support
significantly and offer real packages for all platforms. At that point,
I'll see if this could help with the pycryptopp packages too.

Also, we should probably create a second APT repo: one for the
per-commit development debs, and a separate one that contains only the
release debs. Folks who want to live on the bleeding edge can pull from
the dev repo, but folks who just want to easily automatically update to
the latest official release can point at the relase-repo. I think the
Makefile rules that upload a deb should look at the version number to
decide whether a deb gets uploaded to the release-repo or not.

cheers,
 -Brian