yet another failure of setuptools to set up a correct sys.path

[davidsarah]

​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/534/steps/install-to-egg/logs/stdio

Even though the setup-time requirement for zope.interface in that build was "zope.interface == 3.3.1, == 3.5.3, == 3.6.1", the version that was on the sys.path when building Nevow was 3.6.4.

[davidsarah]

This is getting ridiculous -- we've had dozens of instances where setuptools sets up a sys.path that loads the wrong code, all for subtly different reasons. IMHO, setuptools has to go.

[davidsarah]

Note that the Tahoe version here is r5073, not the 1.8.2 release.

[zooko]

Replying to davidsarah:

This is getting ridiculous -- we've had dozens of instances where setuptools sets up a sys.path that loads the wrong code, all for subtly different reasons. IMHO, setuptools has to go.

Maybe we should open a ticket named "setuptools delenda est", or instead start a mailing list thread. My first question would be: is there reason to believe it would be worth switching to distribute instead? (My instinct: no. The distribute project changed a bunch of things without first writing unit tests, thus resulting in an even buggier codebase than the original setuptools. Then they abandoned (?) it.)

What about the new Distutils2 project? (The successor from the same group that did the aforementioned "distribute" project.) As far as I know it isn't yet supported by the makers of our deps, so for example as far as I understand the pyOpenSSL project doesn't -- or can't? -- produce a package which provides pyOpenSSL on Windows when the user doesn't have a compiler.

Now we could still make our build system build Tahoe-LAFS on all supported operating systems including Windows when the user does have a compiler, and we could still provide a complete pre-built bundle of binaries of pyOpenSSL with Tahoe-LAFS for certain specific operating systems, but this would mean users no longer have the ability to acquire binaries of deps from upstream (e.g. pyOpenSSL maintainers) and source code from Tahoe-LAFS and use the two together.

Also, even to get that far would be a great deal of effort on our part, wouldn't it? Unless Distutils2, or pip, or something is a lot more functional than I thought.

[davidsarah]

Replying to zooko:

Replying to davidsarah:

This is getting ridiculous -- we've had dozens of instances where setuptools sets up a sys.path that loads the wrong code, all for subtly different reasons. IMHO, setuptools has to go.

Maybe we should open a ticket named "setuptools delenda est", or instead start a mailing list thread. My first question would be: is there reason to believe it would be worth switching to distribute instead?

No. distribute doesn't seem to have fixed any of the things in setuptools that I consider most broken, and has introduced its own bugs.

What about the new Distutils2 project? (The successor from the same group that did the aforementioned "distribute" project.) As far as I know it isn't yet supported by the makers of our deps, so for example as far as I understand the pyOpenSSL project doesn't -- or can't? -- produce a package which provides pyOpenSSL on Windows when the user doesn't have a compiler.

If setuptools is used to produce a given pyOpenSSL egg, that doesn't mean you need setuptools to run an application that uses it. That's just a matter of putting the egg in the right position on the sys.path (which setuptools can't do reliably), and then the Python ZipImport machinery will load it, including loading native libraries.

So run-time should be easy, and we shouldn't need the script that setuptools generates in the support directory.

At build-time, we do need to run the setup.py files of dependencies. If we build them in subprocesses, set up the sys.path for each subprocess correctly (i.e. with that package's build-time dependencies), and prevent setuptools from downloading packages, then the scope that setuptools has to mess up each build is much reduced. Then we can incrementally change (or persuade its maintainers to change) each package to not depend on setuptools at build time.

Now we could still make our build system build Tahoe-LAFS on all supported operating systems including Windows when the user does have a compiler, and we could still provide a complete pre-built bundle of binaries of pyOpenSSL with Tahoe-LAFS for certain specific operating systems, but this would mean users no longer have the ability to acquire binaries of deps from upstream (e.g. pyOpenSSL maintainers) and source code from Tahoe-LAFS and use the two together.

No, because if the pyOpenSSL maintainers want to continue to use setuptools to generate their eggs, we're not stopping them. It's our own use of setuptools that we want to replace first, because that's where most of the problems lie.

[davidsarah]

Copy of ​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/534/steps/install-to-egg/logs/stdio/text

[zooko]

Let's move this discussion to its own ticket or the mailing list.

[davidsarah]

The "setuptools delenda est" ticket is #1582. (It doesn't replace this ticket which is documenting a specific problem with setuptools.)

[warner]

We've stopped using eggs, virtualenvs are protecting us against system-installed packages, and modern setuptools is working pretty well for us. I think we can close this now.