#709 closed defect (fixed)

hard to run against alternate dependencies, e.g. trunk version of Foolscap

Reported by: warner Owned by: warner
Priority: major Milestone: 1.11.0
Component: packaging Version: 1.4.1
Keywords: setuptools Cc:
Launchpad Bug: 821000

Description (last modified by warner)

For the #653 work I'm doing now (which requires a change to foolscap: foolscap#105), I need to run a Tahoe trunk tree against a Foolscap trunk tree (to test changes I'm considering putting into the upcoming foolscap release). The normal way to do this (which used to work before we switched to setuptools) was:

PYTHONPATH=~/foolscap/trunk make test

I tried to do this yesterday, and here are the problems I ran into:

  • the Foolscap .egg that was built in support/lib overrides anything else on PYTHONPATH, so I must delete the .egg first
  • on OS-X, there's also an .egg put in the root of the tahoe tree (the #657 problem), so I must delete that one too
  • "make test" takes forever (#591) and rebuilds a Foolscap .egg before running any tests, so I use "make quicktest" instead
  • A Foolscap source tree (in my ~/foolscap/trunk directory) does not appear to qualify as fulfilling the setuptools dependency for foolscap[secure_connections]>=0.3.1. src/allmydata/scripts/runner.py has a line which does pkg_resources.require('allmydata-tahoe'), and that throws a DistributionNotFound exception. I commented that line out.
  • the pkg_resources.load_entry_point-based support/bin/tahoe script does an implicit pkg_resources.require, also failing because it thinks it doesn't have Foolscap available.

I think the only way I can work around this is to use "setup.py develop TARGETDIR" on my Foolscap tree, and then set PYTHONPATH to point at TARGETDIR instead of ~/foolscap/trunk.

This is annoying. I really wish that PYTHONPATH could be used as it was originally intended, and that these eggs weren't getting in the way.

Change History (15)

comment:1 Changed at 2009-05-18T21:24:10Z by warner

  • Description modified (diff)

fix formatting of Foolscap ticket link

comment:2 Changed at 2009-05-18T21:38:24Z by zooko

The first two problems in your list of five problems are http://bugs.python.org/setuptools/issue53 (respect the PYTHONPATH). There have been two different patches posted to that ticket (the first was by me), and I don't currently understand the objection of the setuptools maintainer (PJE) to the most recent patch. Perhaps you could help by reading that ticket and figuring out what needs to change to get PJE to accept Mark Sienkiwicz's patch.

Alternately, we could continue our toothpick of setuptools (which I call zetuptoolz) by applying my patch or Mark Sienkiwicz's patch to zetuptoolz.

For what it is worth, this is going to be fixed one way or another in the long run, as either distutils core or various setuptools competitors move ahead and we will be able to switch to that in order to gain "respect the PYTHONPATH" without losing other functionality that we currently use.

Now, as for workarounds, if you build a .egg-info file, I think it would eliminate the last 3 of your 5 complaints. I'm not sure if that will do it. .egg-info files are created by default by modern Python distutils when building.

comment:3 Changed at 2009-05-18T22:35:23Z by warner

Yup, by doing python setup.py build in the Foolscap tree, it created a foolscap.egg-info directory there, after which PYTHONPATH=~/foolscap/trunk make quicktest started working again (without commenting out the pkg_resources.require calls).

Also, PYTHONPATH=~/tahoe/trunk/support/lib/python2.5/site-packages python setup.py develop --prefix ~/tahoe/trunk/support put a foolscap.egg-link file in the tahoe lib directory, from which tahoe was able to find the foolscap code.

(I'm not used to doing python setup.py build with foolscap, because Foolscap doesn't need it for anything).

This works well enough for now. I look forward to seeing the "respect PYTHONPATH for .eggs" fix get into whatever packaging system we use. I guess I'll leave this ticket open until that happens. Thanks!

comment:4 Changed at 2009-05-20T01:20:13Z by warner

I was mistaken. python setup.py build does not create a foolscap.egg-info directory. You must do a setup.py egg_info to get it to create the .egg-info . The setup.py develop merely creates it as a side-effect.

comment:5 Changed at 2009-09-24T19:58:58Z by cgalvan

Note that once #668 is resolved, switching to Distribute, this issue may or may not be resolved as a result.

comment:6 Changed at 2010-04-04T16:51:21Z by davidsarah

  • Keywords setuptools added
  • Summary changed from hard to run against alternate dependencies, i.e. trunk version of Foolscap to hard to run against alternate dependencies, e.g. trunk version of Foolscap

comment:7 Changed at 2010-04-04T17:08:06Z by zooko

I opened a ticket with the Python Distribute project: http://bitbucket.org/tarek/distribute/issue/136/respect-the-pythonpath

comment:8 Changed at 2010-10-15T06:43:46Z by zooko

  • Keywords test-needed added

comment:9 Changed at 2010-11-30T15:27:00Z by zooko

zetuptoolz has a patch which is intended to fix this: http://tahoe-lafs.org/trac/zetuptoolz/changeset/20081115185932-92b7f-d451b10759897b0fa5d038f2c42cdcd3901289a8/trunk

If you wanted to try it out by installing zetuptoolz instead of your normal setuptools-or-distribute and try PYTHONPATH=~/foolscap/trunk make test, I would appreciate it.

comment:10 Changed at 2010-11-30T17:26:46Z by zooko

Oh, it isn't just installing zetuptoolz -- this code gets written into your easy-install.pth. So to test this, read the patch and make the appropriate adjustment to the code in your easy-install.pth, and then see if PYTHONPATH=~/foolscap/trunk make test works. If so, then maybe we can extend zetuptoolz so that it hunts down all easy-install.pth's that it has write access to and fixes them. ;-)

Here is the code in question: zetuptoolz's easy_install.py

comment:11 Changed at 2010-11-30T17:26:59Z by zooko

  • Owner changed from somebody to warner

comment:12 Changed at 2010-11-30T17:27:09Z by zooko

warner: are you willing to try this?

comment:13 Changed at 2011-08-04T16:27:55Z by zooko

  • Launchpad Bug set to 821000

comment:14 Changed at 2011-08-04T16:28:41Z by zooko

I've opened Launchpad bug #821000 to track the way this affects setuptools, distribute, Ubuntu, and Tahoe-LAFS (via our local fork of setuptools named "zetuptoolz").

comment:15 Changed at 2016-03-26T21:20:54Z by warner

  • Description modified (diff)
  • Keywords test-needed removed
  • Milestone changed from undecided to 1.11.0
  • Resolution set to fixed
  • Status changed from new to closed

This is now resolved: by using a virtualenv, I can pip install --editable ~/my/foolscap to use a locally-modified Foolscap along with my locally-modified Tahoe. It's not $PYTHONPATH manipulation, but it works just as well.

Note: See TracTickets for help on using tickets.