#530 closed enhancement (wontfix)

use setuptools's --multi-version mode

Reported by: zooko Owned by: zooko
Priority: major Milestone: 1.8.1
Component: packaging Version: 1.2.0
Keywords: setuptools Cc:
Launchpad Bug:

Description (last modified by zooko)


This would make it so that if there is already an older version of a dependency, e.g. pyutil, present on the system and we require a newer version, that it would go ahead and install the newer version, and Tahoe would use the newer version, without requiring the user to delete the older version from her system.

Change History (20)

comment:1 Changed at 2008-11-25T20:06:49Z by zooko

  • Description modified (diff)

This was implemented in 0de6e616e0d0890d but it caused a mysterious problem on most buildbots. I think the problem might be due to an interaction with our "fake setuptools" code in our "Trial" class in setup.py and that the problem might be fixed by Chris Galvan's solution to #505 (wanted: a setuptools plugin to make unit tests be executed with trial instead of with pyunit). Anyway, I undid this change in 1d377cc2d9871e39 in order to make the buildbots green again. By the way, this issue seems to be interfering with Francois's attempt to fix #534 ("tahoe cp" command encoding issue) since an older version of simplejson is installed system-wide on the buildslave that he is experimenting with.

comment:2 Changed at 2008-12-07T22:01:43Z by warner

Zooko mentioned, in response to #555 ("tahoe .deb cannot be installed on hardy because of simplejson dependency"), that this #530 ticket "would have meant that simplejson upgrade happened automatically".

I'm not sure I see how --multi-version would help this. We know that the tahoe .deb is not supposed to contain anything but tahoe code. What --multi-version might do is to make it easier to build tahoe from source on a host that has an older version of e.g. simplejson installed from a deb. But that seems to work anyways right now.

comment:3 Changed at 2008-12-08T18:15:05Z by zooko

Oh I guess I confused #555 with the deployment issue that arose this weekend. That issue involved building from source (or, equivalently, running easy_install allmydata-tahoe. This ticket -- #530 -- would have made it so that deployment attempt would have automatically succeeded by automatically downloading and installing a newer simplejson in egg or source form. Which isn't what #555 is requesting.

comment:4 Changed at 2009-01-15T16:32:35Z by zooko

  • Owner changed from somebody to cgalvan

comment:5 follow-up: Changed at 2009-01-20T19:01:40Z by zooko

Sigh, 8148366d93c34dbe unapplies this patch, again, because setuptools doesn't create the easy-install.pth, setuptools.pth, and site.py files when we add --multi-version.

http://bugs.python.org/setuptools/issue57 # develop doesn't create .pth files and site.py if --multi-version

comment:6 in reply to: ↑ 5 Changed at 2009-12-07T03:32:34Z by davidsarah

  • Keywords setuptools added

Replying to zooko:

http://bugs.python.org/setuptools/issue57 # develop doesn't create .pth files and site.py if --multi-version

which got closed with the following comment:

Behavior is as intended. More precisely, it is an intended feature of

--multi-version that it does not require .pth or site.py support in the target directory.

Can someone translate this to English for those unfamiliar with the vagaries of Python package management? Is this just more of the same "setuptools is broken-by-design" that seems to be causing so many other problems?

comment:8 Changed at 2010-10-05T20:28:42Z by zooko@…

  • Resolution set to fixed
  • Status changed from new to closed

In 98ffbfb31faccdaf:

setup: add --multi-version to the "setup.py develop" command-line
fixes #530. I earlier tried this twice (see #530 for history) and then twice rolled it back due to some problems that arose. However, I didn't write down what the problems were in enough detail on the ticket that I can tell today whether those problems are still issues, so here goes the third attempt. (I did write down on the ticket that it would not create site.py or .pth files in the target directory with --multi-version mode, but I didn't explain why *that* was a problem.)

comment:9 Changed at 2010-10-06T18:26:09Z by warner

After this change, a "python setup.py build" that shouldn't do anything (i.e. do it twice and capture the output from the second run) emits 353 lines, including 16 copies of a warning about how to use versioned dependencies in the importing code:

Using /home/warner/stuff/tahoe/darcs-trunk/support/lib/python2.6/site-packages/zbase32-1.1.2-py2.6.egg

Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:

    pkg_resources.require("zbase32")  # latest installed version
    pkg_resources.require("zbase32==1.1.2")  # this exact version
    pkg_resources.require("zbase32>=1.1.2")  # this version or higher

Note also that the installation directory must be on sys.path at runtime for
this to work.  (e.g. by being the application's script directory, by being on
PYTHONPATH, or by being added to sys.path by your code.)

I suppose that this is a benign warning, but there's too much output there for me to tell: I first assumed that it was a fatal error.

After building it, trying to run setup.py test on my debian box failed (after emitting the same 353 lines of warnings):

pkg_resources.VersionConflict: (pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6), Requirement.parse('pycryptopp>=0.5.20'))

That 0.5.17 is from the current debian/sid python-pycryptopp .deb package, which includes .egg-info data (and pkg_resources.require("pycryptopp") returns 0.5.17).

Does --multi-version break if there is a version on PYTHONPATH that was installed without --multi-version?

comment:10 Changed at 2010-10-06T19:34:21Z by warner

Running python -c 'import pkg_resources; print pkg_resources.require("pycryptopp")', either from inside or outside my tahoe tree, reports:

[pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6)]

Amending that to use "pycryptopp>=0.5.20" results in an exception in both places:

64:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("pycryptopp>=0.5.20")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 654, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 556, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6), Requirement.parse('pycryptopp>=0.5.20'))

Adding support/lib/python2.6/site-packages to my PYTHONPATH (which is what I vaguely remember we used to do inside bin/tahoe) doesn't change its behavior. This is the same error I'm seeing when I do setup.py test.

I started with a setup.py build, so my support/lib/python2.6/site-packages directory contains:

55:warner@fluxx% ls support/lib/python2.6/site-packages/
allmydata-tahoe.egg-link  foolscap-0.5.1-py2.6.egg/                  setuptools-0.6c16dev2.egg/
argparse-1.1-py2.6.egg/   pycryptopp-0.5.25-py2.6-linux-x86_64.egg/  zbase32-1.1.2-py2.6.egg/
darcsver-1.6.3.egg/       pyutil-1.7.12-py2.6.egg/                   zfec-1.4.7-py2.6-linux-x86_64.egg/

I wonder if that -linux-x86_64 in there is causing problems.

comment:11 Changed at 2010-10-30T00:29:55Z by davidsarah

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopening until the warnings in comment:9 are fixed.

comment:12 follow-up: Changed at 2010-10-30T04:20:59Z by zooko

Maybe we don't need --multi-version? Now that we have a reliable test of this behavior (see the step "test-with-fake-pkg" on buildbot, or run python misc/build_helpers/test-with-fake-pkg.py), maybe we should try removing --multi-version and see if that test stays green.

comment:13 in reply to: ↑ 12 Changed at 2010-10-30T05:57:27Z by davidsarah

Replying to zooko:

Maybe we don't need --multi-version? Now that we have a reliable test of this behavior (see the step "test-with-fake-pkg" on buildbot, or run python misc/build_helpers/test-with-fake-pkg.py), maybe we should try removing --multi-version and see if that test stays green.


comment:14 Changed at 2010-10-30T05:57:44Z by davidsarah

  • Owner changed from cgalvan to zooko
  • Status changed from reopened to new

comment:15 Changed at 2010-10-30T06:18:13Z by zooko

  • Status changed from new to assigned

comment:16 Changed at 2010-11-15T09:45:41Z by zooko

I created a ticket530 branch to test patches on the buildbot. Then I forced a build on all buildbots with the current ticket530 (which was unchanged from the current trunk), and now I will push a patch to remove --multi-version (rolling back that part of 98ffbfb31faccdaf) to see if it fares better or worse than trunk.

comment:17 Changed at 2010-11-15T09:49:54Z by zooko

Okay I committed 5f61bad92d8766e7 to the source:ticket530 branch.

comment:18 Changed at 2010-11-15T21:03:28Z by zooko

Removing --multi-version didn't induce any regressions on buildbot. I guess I would like to add a unit test making sure that build doesn't emit certain scary warnings, which test will go from red to green when I commit 5f61bad92d8766e7 to trunk...

comment:19 Changed at 2010-11-16T07:33:06Z by zooko

  • Resolution set to wontfix
  • Status changed from assigned to closed

Fixed in 5f61bad92d8766e7. I decided not to add a unit tests as I couldn't think of a way to check whether the output of build was too scary without "overfitting" and just checking for this particular message text.

comment:20 Changed at 2010-11-16T07:34:43Z by zooko

  • Milestone changed from undecided to 1.8.1
Note: See TracTickets for help on using tickets.