unnecessary rebuild of dependencies when tahoe-deps/ is present

[warner]

Using a trunk 9a02f4a1043d6288 tree, on my debian/sid box, performing "python setup.py build" with the tahoe-deps tarball unpacked in the parent directory causes several dependency libraries to be built and installed in support/lib/python2.5/site-packages even though those libraries are already present, usable, and visible to pkg_resources.requires in the standard PYTHONPATH /usr/lib/python2.5/site-packages directory.

The libraries that are unnecessarily rebuilt are:

• foolscap-0.4.1
• twisted-8.2.0
• simplejson-2.0.9

Removing ../tahoe-deps, erasing support/lib, and re-running "python setup.py build" correctly skips the build of these libraries, resulting in a support/lib that only contains setuptools-0.6c12dev and the allmydata-tahoe.egg-link file (well, and the usual setuptools noise: setup.py, setuptools.pth, and easy-install.pth).

Something in setuptools is buggy: having ../tahoe-deps available should not influence the decision to install a library or not.

This needs to be added to the setuptools bugtracker at ​http://bugs.python.org/setuptools/ .

[zooko]

Thank you for the bug report. As I'm sure you know, a good way to report this bug to setuptools devs would be to create a minimal test case. Here's a crack at one:

Make a setup.py file in a new empty directory with the following contents:

from setuptools import setup
setup(name="minbug", install_requires="simplejson>=1.4", find_links="../tahoe-deps")

Then run python ./setup.py build, with and without a simplejson-2.0.9.tar.gz located in ../tahoe-deps.

[zooko]

This may be related to ​http://bugs.python.org/setuptools/issue17 (easy_install will install a package that is already there). There is a work-around documented in that ticket.

[warner]

I'm not sure it is related.. as I understand it, setuptools-17 is about setuptools building packages that are present in an unusual directory that the platform has added to sys.path (like the /var/lib/python-support/python2.5 on debian), probably because it isn't honoring the platform's changes to sys.path when it scans for those packages.

In contrast, what I observed was that all of the unecessarily-rebuilt libraries were present in /usr/lib/python2.5/site-packages . None were in /var/lib/python-support/python2.5 .

[zooko]

This may be related to #657 ("python ./setup.py test" rebuilds packages).

[cgalvan]

Note that once #668 is resolved, switching to Distribute, this issue may or may not be resolved as a result.

[zooko]

It looks like this may be related to ​http://bugs.python.org/setuptools/issue65 (setuptools bug,inconsistent replacement of eggs when installing from ​file:// urls.) which is fixed in setuptools-0.6c10 prerelease. setuptools 65 is related to ​http://bugs.python.org/setuptools/issue17 (easy_install will install a package that is already there) and ​http://bugs.python.org/setuptools/issue20 (package required at build time seems to be not fully present at install time?).

[zooko]

Maybe someone (cgalvan?) could test this issue with the setuptools v0.6c10.

[zooko]

There was a different bug which caused a similar behavior in #229.

[zooko]

In comment:7:ticket:1168 and comment:10:ticket:1168 we commented on this issue. David-Sarah wrote:

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

and I wrote:

Looks like it might be #717. Isn't the buildbot already testing this by running the "Desert Island build"? e.g: ​http://tahoe-lafs.org/buildbot/builders/clean/builds/2522

In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:8)?

Makefile

[davidsarah]

Replying to zooko:

[...] Isn't the buildbot already testing this by running the "Desert Island build"? e.g: ​http://tahoe-lafs.org/buildbot/builders/clean/builds/2522

In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:8)?

In my scenario, I download a non-SUMO tarball, run python setup.py trial python setup.py build followed by python setup.py trial (expecting the dependencies to be downloaded and built before the test), and then run python setup.py trial again (expecting the previously downloaded dependencies to be used and not rebuilt). Are these expectations correct?

In any case, this is not the same thing as downloading tahoe-deps.tar.gz, untarring it in the distribution directory and then running python setup.py build, which is what test-desert-island does.

[warner]

We no longer provide or support the tahoe-deps tarball/directory, and the pip http+wheel cache works correctly to avoid rebuilds. Closing this out.