prefer binary eggs which satisfy dependencies over source eggs which are even newer

[zooko]

Currently if the zetuptoolz-based build system finds both a binary egg (for the current platform that it is building on) and a source egg, and both of them satisfy the dependency requirement, it will choose the one with the highest version number.

This is a serious problem in practice because, for example, if we have built and hosted dozens of binary eggs of zfec for many different combinations of platform and Python version, and then a newer source-code distribution of zfec is uploaded to PyPI, then the build system will no longer try to use any of those binary eggs and instead use the newer source distribution exclusively, which will fail on platforms that don't have a C compiler and Python header files present.

To fix this, change the logic in zetuptoolz to prefer a binary egg, provided that it is for your platform and Python version and satisfies the dependency requirement, over a source distribution.

[zooko]

comment on a different ticket about one of the (very numerous) times that this has caused trouble for a user:

​http://tahoe-lafs.org/trac/tahoe-lafs/ticket/773#comment:7

bug report against setuptools:

​http://bugs.python.org/setuptools/issue83

[zooko]

David-Sarah says we shouldn't try changing setuptools/zetuptoolz at the last minute, so instead of doing this ticket for v1.8.1, we're just going to do the time-consuming but predictable work of building and hosting binary eggs for all platforms that we want to support.

[davidsarah]

Replying to zooko:

David-Sarah says we shouldn't try changing setuptools/zetuptoolz at the last minute, so instead of doing this ticket for v1.8.1, we're just going to do the time-consuming but predictable work of building and hosting binary eggs for all platforms that we want to support.

Having seen ​the scale of the problem in generating all those binary eggs, zooko and I have had second thoughts about that. I have a patch to zetuptoolz, and zooko has a possible test.

[davidsarah]

bundled zetuptools: prefer platform-specific dists to platform-independent ones. refs #1233

[david-sarah@…]

In 306ec02c1d2946b2:

setup.py: fix a bug in the check for whether we are running 'trial' or 'test', that affected zooko's test for #1233. refs #1233

[zooko]

There is now a deterministic test of this behavior, added in 59dcd5014dcb440a. You can run it locally with python misc/build_helpers/test-with-fake-dists.py. Unfortunately it just skips the test if a satisfying version of pycryptopp is already installed. I couldn't figure out how to test it in that case.

If a satisfying version of pycryptopp is not already installed, then it will test this behavior and either raise an exception saying that the build system attempted to build a fake "pycryptopp v9.9.99" from source package (so it failed the test) or it will pass saying that it used a fake "pycryptopp 0.5.24" from binary package.

Looking at our buildslaves, all of ones working well enough to run the test already have pycryptopp installed:

• ​http://tahoe-lafs.org/buildbot/builders/David%20A.%20OpenSolaris%20i386/builds/464/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/MM%20netbsd5%20i386%20warp/builds/159/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/lucid-amd64/builds/72/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/63/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Ruben%20Fedora/builds/691/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/356/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/213/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/356/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/168/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/671/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Randy%20FreeBSD-amd64/builds/195/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Arthur%20lenny%20c7%2032bit/builds/553/steps/test-with-fake-dists/logs/stdio

I just manually removed pycryptopp from zomp and rebuilt build #213 to yield build #214:

• ​http://tahoe-lafs.org/buildbot/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/214/steps/test-with-fake-dists/logs/stdio

Hey! Same behavior. Oh, silly me! pycryptopp is being built and stored in ./support by the build step if it isn't already present, so when the later test-with-fake-dists step happens it is always there! I will change the buildmaster config to do with-with-fake-dists first and see if that shows us something useful.

[zooko]

Great! Some of the buildslaves don't have pycryptopp installed and so we get a demonstration of issue #1233 on them. If we commit a patch that fixes #1233 then all of these should go from red to green:

• ​http://tahoe-lafs.org/buildbot/builders/lucid-amd64/builds/74/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/65/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Randy%20FreeBSD-amd64/builds/197/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20CentOS5-i386/builds/232/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/358/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/216/steps/test-with-fake-dists/logs/stdio

These ones have a sufficient version of pycryptopp and so are not really testing #1233:

• ​http://tahoe-lafs.org/buildbot/builders/Ruben%20Fedora/builds/693/steps/test-with-fake-dists/logs/stdio
• ​http://tahoe-lafs.org/buildbot/builders/MM%20netbsd5%20i386%20warp/builds/161/steps/test-with-fake-dists/logs/stdio

These ones I went to bed before they finished so at the time of this writing I don't knowL

• ​http://tahoe-lafs.org/buildbot/builders/David%20A.%20OpenSolaris%20i386/builds/467
• ​http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/673
• ​http://tahoe-lafs.org/buildbot/builders/Arthur%20lenny%20c7%2032bit/builds/555

[zooko]

All of the red steps test-with-fake-dists on our ​Supported Builders or ​all builders should go green when we fix this.

[zooko]

It would be great if someone else would try to hack zetuptoolz to fix this. If they don't, I will try.

[zooko@…]

In [4832/ticket1233]:

bundled zetuptoolz: prefer locally-available distributions over remotely-downloaded distributions above all
This fixes #1233. Actually the previous patches—a1cef915fd689308 and a44330dfeef23757—fixed it, but with them zetuptoolz would download a higher-numbered distribution from the net instead of using the locally-available (fake) pycryptopp-0.5.24, thus preventing the tests from passing. This patch changes that behavior (which is an improvement in its own right) and also fixes a bug in the tests.

[zooko]

This is fixed by a1cef915fd689308, a44330dfeef23757, b4c14421f7c2f25a, 8fef75851181ce79 on the ticket1233 branch! Compare ​before and ​after those four patches.

These patches also make a related improvement to behavior: do not download a distribution from a remote server if a distribution already present locally would suffice.

Please review! :-)

[zooko]

By the way, here is the issue ticket in the Distribute project:

​http://bitbucket.org/tarek/distribute/issue/127/prefer-v141-in-binary-egg-format-over-v145

[davidsarah]

Looks like there are still some unexplained buildslave failures on this step, so not ready for review yet.

[davidsarah]

Replying to zooko:

These patches also make a related improvement to behavior: do not download a distribution from a remote server if a distribution already present locally would suffice.

However, they always look on remote servers to see what distributions are available, before choosing the local one. It shouldn't be necessary even to look remotely in that case. But fixing that would be part of #1220 (or the longer-term project of replacing our use of setuptools).

[zooko]

Replying to davidsarah:

Looks like there are still some unexplained buildslave failures on this step, so not ready for review yet.

Let's see...

• ​http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/78
• ​http://tahoe-lafs.org/buildbot/builders/Shawn%20ubuntu-amd64/builds/592/steps/test-with-fake-dists/logs/stdio

Both of them exhibit the same error message:

pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/lib/pymodules/python2.6), Requirement.parse('foolscap[secure_connections]>=0.5.1'))

and

pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/local/lib/python2.6/dist-packages/foolscap-0.4.2-py2.6.egg), Requirement.parse('foolscap[secure_connections]>=0.5.1'))

Now what do these two have in common that the rest of the buildslaves, including their brother ​lucid-amd64 don't?

The answer seems to be that they have foolscap-0.4.2 installed, as shown by flappclient --version in their show-tool-versions steps. The other buildslaves, which don't hit this error, either have no foolscap installed at all or have a foolscap installed which is >= 0.5.1.

Still investigating...

[zooko]

Hm, that VersionConflict looks kind of like the problem in #1190. Could that have regressed somehow but only with regard to foolscap? Could it have something to do with the "extra requirement" [secure_connections] that aren't used in our other dependencies? That fix was suppoesd to be verfied by these tests: 390c40cd8ce1e579, [0101030034303-93fa1-0267f4ed3c9014099eb2c9b4d8e43724d7ff85e1], c3b1bca95450ae40.

Huh-oh, if I change the test suite that trunk/misc/build_helpers/test-with-fake-pkg.py@4801 uses to test from allmydata.test.test_base62 (chosen because it does little) to allmydata.test.test_backupdb (which imports pycryptopp) then test-with-fake-pkg.py fails! On my Mac OS 10.6. This is a regression, because it worked back when we committed it to trunk!

I'll attach a darcs patch which changes test-with-fake-pkg to use test_backupdb...

[zooko]

Okay let's move further discussion of this VersionConflict over to ​http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1190#comment:38 .

[zooko]

I kind of suspect that the problems with the VersionConflict aren't actually caused by the patches in this ticket, but are somehow being triggered by the test-with-fake-dists step. Perhaps we don't see this VersionConflict on buildbot of current trunk merely because the test-with-fake-dists step ends early on trunk when it gets an exception from trying to build the fake pycryptopp-9.9.99. If we, for example, hacked the fake pycryptopp-9.9.99 so that it stopped raising an exception, thus allowing test-with-fake-dists to run to the end on trunk, would we get the same VersionConflict on "Brian ubuntu-i386 linode" and "Shawn ubuntu-amd64"?

[zooko]

The issue with the VersionConflict with foolscap is fixed by 50f8c37a2b0049a5 (on ticket1233).

The buildbot is nicely green now with &branch=ticket1233:

​http://tahoe-lafs.org/buildbot/waterfall?branch=ticket1233

[zooko]

Please review!

[zooko]

On IRC, David-Sarah said: "I think ticket1233 can be committed to trunk"

[david-sarah@…]

In a1cef915fd689308:

bundled zetuptools: prefer platform-specific dists to platform-independent ones. refs #1233

[zooko@…]

In b4c14421f7c2f25a:

bundled zetuptoolz: prefer locally-available distributions over remotely-downloaded distributions above all
This fixes #1233. Actually the previous patches—a1cef915fd689308 and a44330dfeef23757—fixed it, but with them zetuptoolz would download a higher-numbered distribution from the net instead of using the locally-available (fake) pycryptopp-0.5.24, thus preventing the tests from passing. This patch changes that behavior (which is an improvement in its own right) and also fixes a bug in the tests.

[zooko]

This appears to have failed for David-Sarah just now. They reported:

$ python setup.py build
running darcsver
setup.py darcsver: wrote '1.8.1-r4867' into src/allmydata/_version.py
running develop
Not found: ../tahoe-deps
running egg_info
writing requirements to src\allmydata_tahoe.egg-info\requires.txt
writing src\allmydata_tahoe.egg-info\PKG-INFO
writing top-level names to src\allmydata_tahoe.egg-info\top_level.txt
writing dependency_links to src\allmydata_tahoe.egg-info\dependency_links.txt
writing entry points to src\allmydata_tahoe.egg-info\entry_points.txt
writing manifest file 'src\allmydata_tahoe.egg-info\SOURCES.txt'
running build_ext
Processing darcsver-1.7.1.egg
removing 'd:\tahoe\tahoe-clean\support\lib\site-packages\darcsver-1.7.1.egg' (and everything under it)
Copying darcsver-1.7.1.egg to d:\tahoe\tahoe-clean\support\lib\site-packages
darcsver 1.7.1 is already the active version in easy-install.pth
Deleting support/Scripts\darcsver
Deleting support/Scripts\darcsver.pyscript
Installing darcsver.pyscript script to support/Scripts
Installing darcsver script to support/Scripts

Installed d:\tahoe\tahoe-clean\support\lib\site-packages\darcsver-1.7.1.egg
Creating d:\tahoe\tahoe-clean\support\lib\site-packages\allmydata-tahoe.egg-link (link to src)
allmydata-tahoe 1.8.1-r4867 is already the active version in easy-install.pth
Deleting support/Scripts\tahoe
Deleting support/Scripts\tahoe.pyscript
Installing tahoe.pyscript script to support/Scripts
Installing tahoe script to support/Scripts

Installed d:\tahoe\tahoe-clean\src
Processing dependencies for allmydata-tahoe==1.8.1-r4867
Searching for pycryptopp>=0.5.20
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Best match: pycryptopp 0.5.25
Processing pycryptopp-0.5.25.tar.gz
Running pycryptopp-0.5.25\setup.py -q bdist_egg --dist-dir c:\users\david\appdata\local\temp\easy_install-ytavvv\pycrypt
opp-0.5.25\egg-dist-tmp-_ox2l8

Setting up environment to run scripts for the current user...
Already done: associate the filetype .pyscript with Python.File for the current user.
Already done: add .pyscript to the user environment variable PATHEXT.
Already done: add .pyw to the user environment variable PATHEXT.
/usr/lib/gcc/i686-pc-mingw32/3.4.4/../../../../i686-pc-mingw32/bin/ld: cannot find -lmsvcr90
collect2: ld returned 1 exit status
error: Setup script exited with error: command 'g++' failed with exit status 1

I don't understand why it didn't choose ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/pycryptopp-0.5.25-py2.6-win32.egg over pycryptopp-0.5.25.tar.gz . Hm, and from where did it get that pycryptopp-0.5.25.tar.gz , anyway? It doesn't say that it downloaded it. Oh, maybe the logic to "prefer locally available over remote as the highest priority" chose pycryptopp-0.5.25.tar.gz because it was present on the local filesystem.

In that case, perhaps this is the expected behavior and we can reclose this ticket? :-/

Maybe we should patch zetuptoolz to verbosely explain why it chose what it did and where it acquired it?

[davidsarah]

Replying to zooko: [...]

I don't understand why it didn't choose ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/pycryptopp-0.5.25-py2.6-win32.egg over pycryptopp-0.5.25.tar.gz . Hm, and from where did it get that pycryptopp-0.5.25.tar.gz , anyway? It doesn't say that it downloaded it.

Oh, Problem Between Keyboard and Chair. I had that in my tahoe-deps directory.

Oh, maybe the logic to "prefer locally available over remote as the highest priority" chose pycryptopp-0.5.25.tar.gz because it was present on the local filesystem.

In that case, perhaps this is the expected behavior and we can reclose this ticket? :-/

Yes. (It wasn't reopened.)

Maybe we should patch zetuptoolz to verbosely explain why it chose what it did and where it acquired it?

That would be useful, as a new ticket.