Debian etch package size mismatch

[swillden]

The r3571 build of Tahoe 1.2.0 in the allmydata.org debian repository has a size mismatch. The Packages file reports the size as 738,742 bytes, but the .deb file is actually 738,826 bytes in size. This causes apt to refuse to install the downloaded package.

[warner]

argg. What happens is that the Packages file was being rebuilt while a new .deb was being uploaded. This happens because we've got half a dozen different platforms, each of which uploads a .deb and then triggers the Packages-rebuild.. there's a lock in place to prevent overlapping rebuilds, but there's nothing to prevent the rebuild from seeing a partially-uploaded .deb and remembering its (truncated) size forever.

When this happens, we usually manually delete and rebuild the Packages file. But it happens a lot, and is really annoying.

One fix would be to use an uploader which keeps the file hidden (at least named something other than .deb) until the upload is finished. At the moment we're using scp or rsync (I forget which), which does not use a .tmp file. I'd like to change this to use the foolscap "xfer-client", to reduce the authority of the buildslave, and I'd also change xfer-client to use a .tmp file.

I'll rebuild the Packages file today, and then perhaps next week I'll get around to changing the upload process.

[warner]

I'm working on ​foolscap#82, an "app server", which will be the clean way to fix this: buildslaves will get a single-purpose file-uploader FURL, and the server-side receiver will atomically move the inbound .deb files into place, not giving the update-index process a chance to see a partial file.

I'm getting close to finished.. I hope to have a new foolscap release out in the next few days, and then update the upload-deb process another day or two later.

[warner]

Ok, foolscap-0.4.2 is out, and I've installed the new flappclient on all of our deb-producing machines except the feisty slave (I'm having ssh problems today). I've sent email to the admins of the non-allmydata deb-producing machines:

• lenny-amd64: Eugen Leitl
• hardy-amd64: Zandr (fieldcircus)
• intrepid-amd64: Shawn Willden (might really be jaunty-amd64)

and I've set up the "flappserver" on the APT repository side.

Next step is to modify the buildmaster config to use flappclient upload-file to upload debs, at least on a few buildslaves, and see if it works.

[warner]

Oh, another TODO item: replace the OS-X-10.5 xfer-server on hanford with this same flappserver. It's currently dropping Mac .dmg files into ~/public_html/dist/tahoe/mac-10.5/

[warner]

Ok, we're now using flappclient to upload debs for all of our deb-producing machines (including feisty), and a quick test seems to work:

• etch
• edgy
• feisty
• gutsy
• hardy

I'm waiting on responses from the non-AMD machines, with some folks saying they'll take care of it tonight:

• lenny-amd64
• hardy-amd64
• jaunty-amd64

[warner]

note to self, the ubuntu "watershed" package appears like it might handle the locking/idempotent-execution stuff I need to the APT-index-update step.

[zooko]

Brian posted an update on this issue to the list:

​http://allmydata.org/pipermail/tahoe-dev/2009-August/002547.html

[zooko]

Okay now all .deb uploader builders are using flappclient. I guess this ticket is still open because of idea of using "watershed" to lock for the apt-index-update?

[davidsarah]

This ticket appears to be long-obsolete. Brian: please open a new ticket about using watershed if that is still relevant.