setuptools fails to be satisfied with already installed Nevow and tries to install a new one

[zooko]

It is supposed to be the case that you can install the "MANUAL DEPENDENCIES", download the tahoe source distribution, move to a desert island with no Internet, follow the README, and get a working version of Tahoe.

To that end, we bundle source tarballs of the "AUTOMATIC DEPENDENCIES", and we configure our setup process to use those locally available tarballs instead of fetching things from the Internet.

However, this feature is not working. Our setup process keeps fetching things from the Internet even when it doesn't have to, and worse builds fail (sometimes? On some platforms?) if it is unable to download packages.

See also #212 "easy_install of a fat binary .egg which was built on Mac OS 10.4 fails on Mac OS 10.5", and #164 "bundled dep tarballs not used if... if some of the .egg's are already found on the system?".

[zooko]

The setuptools_darcs plugin is often a problem. We need to fix this, by either bundling that package and making the use-bundled-packages feature work, or by no longer requiring setuptools_darcs. This plugin is actually needed only for producing source packages from source -- not for producing binary packages nor for installing locally. The reason that it is required by default is that if someone does get a copy of our source tree and then tries to use it to produce a new source tree (i.e. by running ./setup.py sdist, then without the setuptools_darcs plugin they will get a bad (incomplete) source package with no warning or error message. In order to minimize the chance that somebody stubs their toe on that corner case, I have made the setuptools_darcs plugin mandatory.

[zooko]

fixed by some subset of the patches between 3028defb7e1f00f3 and bfc69178b2f07477 (inclusive on both ends).

[warner]

I'm seeing this problem occur again, on my home machine running debian/unstable. After I upgraded to the debian packaging of 0.6c8 (from 0.6c7), the tahoe build process started downloading and building a Nevow tarball, even though there's a local copy already installed (with a valid-looking .egg-info file) on sys.path .

​http://allmydata.org/buildbot/builders/speed-DSL/builds/315/steps/compile/logs/stdio is an example of the build process with setuptools-0.6c8, in which Nevow is downloaded from divmod.org and compiled.

​http://allmydata.org/buildbot/builders/speed-DSL/builds/312/steps/compile/logs/stdio is an example of a build which failed because the local debian package of nevow-0.9.29 was detected, which locked the target version at 0.9.29, and setuptools then tried to download an 0.9.29 tarball (which failed because by that point divmod.org was hosting a newer version, and so I presume they had removed the old tarball).

​http://allmydata.org/buildbot/builders/speed-DSL/builds/311/steps/compile/logs/stdio is the first build with the new setuptools-0.6c8, doing the extra download-and-build step.

​http://allmydata.org/buildbot/builders/speed-DSL/builds/310/steps/compile/logs/stdio is the last build with the old setuptools-0.6c7, in which it correctly refrains from downloading or building Nevow.

[zooko]

Here's a test which could show whether setuptools/easy_install is working on your system:

Find a Python package which is already installed, such as Nevow.

Now run sudo easy_install nevow. If it downloads anything, then your easy_install is broken.

Alternately, you could get minproj-2.0.0.tar.gz. This is a project that has install_requires=["nevow"] in it. If running sudo ./setup.py install downloads anything, then your setuptools is broken.

You can do this without sudo, by setting the PYTHONPATH and using --prefix...

[warner]

On my sid workstation ('fluxx'), I did the following commands, and observed setuptools download nevow from divmod.org, despite nevow-0.9.31 already being installed such that 'import nevow' worked inside a python shell.

HERE = ~/trees/tahoe-mutable-205-312/t

go:
	mkdir -p $(HERE)/tmp/lib/python2.4/site-packages
	PYTHONPATH=$(HERE)/tmp/lib/python2.4/site-packages easy_install --prefix=$(HERE)/tmp nevow

I will see if I can find a machine on which this same test does *not* download anything, and then compare the setuptools/easy_install installations.

[zooko]

Here's a related question: is Nevow installed on fluxx in such a way that pkg_resources recognizes it and its version number? Try this:

python -c 'import pkg_resources;print pkg_resources.require("nevow")'

and

python -c 'import pkg_resources;print pkg_resources.require("nevow == 0.9.31")'

[warner]

yes, I believe so:

1293:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("nevow")'
[Nevow 0.9.31 (/var/lib/python-support/python2.4)]
1294:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("nevow == 0.9.31")'
[Nevow 0.9.31 (/var/lib/python-support/python2.4)]

[zooko]

This has been ticketed on ​the setuptool s issue tracker issue17 and ​debian bug report 475440.

[warner]

For reference, this is still broken on my debian/sid system: Nevow-0.9.31 is downloaded and built even though pkg_resources.require("nevow") indicates that 0.9.31 is available in /var/lib/python-support/python2.5 .

[warner]

we're also seeing this on a hardy box

[zooko]

Confirmed that this problem happens on our new Hardy server, and also I confirmed that it stops happening when you uninstall the python-setuptools package from Ubuntu and instead install setuptools from upstream directly. I'll update the Debian bug report and open a sibling bug report for Ubuntu...

[zooko]

Added a bug report for Ubuntu:

​https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035

[zooko]

Ah, it turns out that this is not caused by the Debian/Ubuntu? packaging! It happens also with the upstream setuptools. I'm not sure why I thought it didn't (as earlier posted in this ticket).

[warner]

still happening on a hardy box (deharo2). My workaround was to uninstall tahoe+nevow, thus allowing the buildslave to download+compile nevow on each build.

[zooko]

By the way, I suspect that #455 (setuptools sandbox isn't tight enough to hold Twisted's sand in so that it doesn't get in Nevow's eyes) might be an underlying cause of this, or at least might be related.

[warner]

The same problem was causing build failures via python-openssl, since 0.6 is not being distributed anymore (or it's been removed from certain sf.net mirrors, or sf.net is having connectivity problems). In ae6cc14a2da9d7f1 I added a workaround: just add --site-dirs=/var/lib/python-support/python2.5/ to the 'setup.py develop' command line. This should fix the problem on debian-like systems and not cause any ill effects on other platforms.

I expect that this ought to cause our buildslaves to stop trying to build nevow and openssl when they are already installed.

[robk]

so I had a very similar problem today, on the mac

it turned out that easy_install had at some point installed twisted into /Library, and somehow tahoe (specifcally make test) was pulling in twisted from /System/Library? (the apple supplied version) which wound up barfing on an error it got whilst cross-importing bits of twisted from out of /Library. I removed the twisted egg etc from /Library, but also needed to add

"--site-dirs /System/Library/Frameworks/Python?.framework/Versions/2.5/Extras/lib/python"

to the build args in order to get tahoe's build to not try and install a different version of twisted and instead use the extant system one. I'll submit a patch to that end shortly.

[warner]

Note that we had to add a --site-dirs argument to get setuptools to recognize the python modules that get installed (on debian) into a /var/lib directory that is on sys.path but isn't one of the traditional places.

Take a look at setup.py for the place to add it. It should be harmless to add another one.

[zooko]

Well, I guess this is fixed by our --site-dirs hack. Not very satisfying (I would really prefer it if setuptools would search the sys.path for requirements per ​http://bugs.python.org/setuptools/issue17 ), but the allmydata-tahoe-relevant portion of this issue is fixed. Right?

[launchpad]

Updating Launchpad bug reference

[zooko]

This was marked as fixed long ago, possibly in 1.2.0 but definitely in 1.3.0. Note that there is currently an observed bug on some platforms which smells like this one, so perhaps this should be reopened. (On Zandr's NAS device and on Brian's Debian system, pycryptopp gets built both during python ./setup.py build and during python ./setup.py test. Okay, that's actually a different underlying issue so I'll open a new ticket for that.