make setup.py more easily patchable by OS packagers

[davidsarah]

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs or by deciding to use a different version of the dependency than would be provided by an OS package.

The attached diff adds an external_dependency_management switch to setup.py to cause this change in behaviour. It doesn't remove the need for a patch, but makes such patches much more likely to apply cleanly.

[davidsarah]

add 'external_dependency_management' flag to setup.py (context diff)

[zooko]

For install requirements you can turn off dependency management by passing --single-version-externally-managed or --root=targetdir options to python setup.py install. For build-time dependencies, they are bundled with the Tahoe-LAFS source tree so you probably don't need to disable the dependency on them.

[zooko]

Replying to davidsarah:

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs

Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present.

or by deciding to use a different version of the dependency than would be provided by an OS package.

Does this mean the packager wants to supply a version of a dependent library older than the minimum version that we specify in _auto_deps.py? If so, I would like to know about this!

[zooko]

I chatted with rockstar on IRC about his packaging of Tahoe-LAFS for Ubuntu. He said that python setup.py test failed due to it trying to execute darcsver and darcsver not being found, which he solved by removing the aliases section of setup.cfg which makes the darcsver command be invoked when running the test command.

I don't understand how darcsver could not been found and caused the test to fail, but perhaps I don't understand something about the build environment (which is managed by debhelper). Here is an example of python setup.py test working on an Ubuntu buildslave: ​http://tahoe-lafs.org/buildbot/builders/Shawn%20jaunty%20amd64/builds/532/steps/test/logs/stdio .

I think the next step is to set up automated execution of a build as managed by debhelper on a buildslave.

Also I would very much appreciate it if someone who knows how to use debhelper would give me a paste of what the error message is when running python setup.py test.

[zooko]

​http://tahoe-lafs.org/pipermail/tahoe-dev/2010-August/004964.html

[davidsarah]

Replying to zooko:

Replying to davidsarah:

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs

Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present.

I've seen it attempt to do so (for foolscap-0.5.1, for example). It's possible that the dependencies were not present in a way recognized by setuptools, but that seems to be a situation that is very easy to end up in. Also see #717 (which is about rebuilding rather than downloading, but that's still a problem).

or by deciding to use a different version of the dependency than would be provided by an OS package.

Does this mean the packager wants to supply a version of a dependent library older than the minimum version that we specify in _auto_deps.py? If so, I would like to know about this!

setuptools might want to download, build, or install a version that is newer than the one supplied by the OS package manager, even though the latter satisfies our declared requirement. (I'm skeptical that ​http://bugs.python.org/setuptools/issue17 and ​http://bugs.python.org/setuptools/issue20 have actually been fixed, or if they have, then there must be other bugs causing this problem, because I've definitely seen it happen.)

[davidsarah]

Replying to davidsarah:

Replying to zooko:

Replying to davidsarah:

Packaging Tahoe-LAFS for OS distributions (e.g. Debian or Ubuntu) currently requires that setup.py be patched, so that setuptools doesn't try to satisfy Tahoe's dependencies by downloading eggs

Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present.

I've seen it attempt to do so (for foolscap-0.5.1, for example).

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

[davidsarah]

Replying to davidsarah:

Replying to davidsarah:

Replying to zooko:

Are we sure that this is a problem? It won't try to download eggs if the dependencies required are already present.

I've seen it attempt to do so (for foolscap-0.5.1, for example).

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

This may be the same problem as #657 (modulo trial vs test, but trial should not be doing any more than test, I think).

[zooko]

Replying to davidsarah:

I can reproduce this by downloading the 1.8.0c2 tarball, building it, and then running python setup.py trial. foolscap 0.5.1 is downloaded again on every run of trial, even though python setup.py build copied it into support/site-packages and the build completed successfully.

Okay let's open a ticket and write a deterministic unit test which shows this misbehavior.

[zooko]

Looks like it might be #717. Isn't the buildbot already testing this by running the "Desert Island build"? e.g: ​http://tahoe-lafs.org/buildbot/builders/clean/builds/2522

In that test, the test code installed the deps tarball and then ran build, and made sure that the code under test (which is our build system) did not try to download any package. The code under test passed and the test is green! So why did it do the right thing and pass that test even if it reproducible does the wrong thing in David-Sarah's scenario (comment:8)?

Makefile

[zooko]

Moving the discussion about re-downloading foolscap to #717.

[zooko]

I think this is a duplicate of #1220 (build/install should be able to refrain from getting dependencies). Please re-open this ticket if I'm missing something.

[davidsarah]

Replying to zooko:

I think this is a duplicate of #1220 (build/install should be able to refrain from getting dependencies). Please re-open this ticket if I'm missing something.

This ticket has a patch specifically to reduce the need for patching setup.py, and that patch got forgotten when it was closed. My interpretation of #1220 is that it's about longer-term fixes.

[warner]

I think our new setup.py now meets this requirement. At least, it behaves like all other setup.pys out there. The lack of setup_requires= is a big one, and using tox for tests means we no longer have custom setup.py test steps (with dependencies).

Closing this one out, but feel free to re-open if I've misinterpreted it.