setuptools downloads and builds a correct version of a dependency in the install-to-egg step, but then adds a different version not satisfying the requirement to easy_install.pth

[davidsarah]

In the attached install-to-egg build on the 'Eugen lenny' buildslave, the requirement for Twisted is >= 10.1.0.

setuptools downloads and builds Twisted 10.2:

Searching for Twisted>=10.1.0
Best match: Twisted 10.2.0
Downloading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/Twisted-10.2.0.tar.bz2
Processing Twisted-10.2.0.tar.bz2
Running Twisted-10.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QTimkw/Twisted-10.2.0/egg-dist-tmp-yBXrHn
[...]

but then it adds Twisted 8.1 to easy-install.pth, resulting in a failed Tahoe build:

Adding Twisted 8.1.0 to easy-install.pth file
[...]
Installed /home/tahoebuildslave/tahoebuildslave/Eugen lenny-amd64/build/egginstalldir/Twisted-8.1.0-py2.5-linux-x86_64.egg
error: Could not find required distribution Twisted>=10.1.0

Since we're installing to an egg rather than globally, there's no good reason why that shouldn't work even if Twisted 8.1 is installed single-version-externally-managed. But if that's not supported for some reason, it's still incorrect to build 10.2 and then fail with a misleading error message.

[davidsarah]

Copy of ​http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/850/steps/install-to-egg/logs/stdio/text

[zooko]

According to ​http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/860/steps/show-tool-versions/logs/stdio , that is setuptools-0.6c8 doing that. That is a very old version. Perhaps this is one of the bugs that has been fixed between setuptools-0.6c8 and the current setuptools-0.6c11 or distribute-0.6.19. We don't see the same bug on other builders, that have newer version of setuptools installed.

I'll ask Eugen in email if he could upgrade the setuptools on there. If not, we could try to reproduce this by installing setuptools-0.6c8, reproducing this failure, then upgrading setuptools to 0.6c11 or distribute-0.6.19 and seeing if the failure goes away.

[zooko]

Eugen upgraded to setuptools-0.6c12dev-r88846 (​show-tool-versions), and this problem went away: ​test-from-egg stdio.

Now there is another problem in the subsequent ​test-from-egg step. The sys.path has /usr/lib/python2.5/site-packages earlier than /home/tahoebuildslave/tahoebuildslave/Eugen lenny-amd64/build/egginstalldir/Twisted-11.0.0-py2.5-linux-x86_64.egg, so Twisted 8.1 (which is in /usr/lib/python2.5/site-packages gets imported and then our check_all_requirements()​ function correctly objects that we imported a disallowed version.

I don't know why /usr/lib/python2.5/site-packages is earlier than the Twisted egg. Setuptools normally puts all .egg's earlier on the sys.path earlier than all non-egg entries (which is the cause of the most widely hated setuptools issue: ​setuptools issue #53 -- respect the PYTHONPATH).

[daira]

Replying to zooko:

I don't know why /usr/lib/python2.5/site-packages is earlier than the Twisted egg. Setuptools normally puts all .egg's earlier on the sys.path earlier than all non-egg entries (which is the cause of the most widely hated setuptools issue: ​setuptools issue #53 -- respect the PYTHONPATH).

setuptools does not always put .egg entries earlier than non-egg entries. If a non-egg is needed to satisfy a dependency and it is in a shared package directory (such as a site-packages directory, either the global one or one in a virtualenv), then other packages from that directory can end up being found before the ones required by Tahoe, even when setuptools thinks theose requirements have been correctly satisfied (see ticket:1246#comment:6). This behaviour is nondeterministic.

[warner]

We no longer use eggs, and pip installs our dependencies in an unpacked form. I think we can close this now.