hard to run against alternate dependencies, e.g. trunk version of Foolscap

[warner]

For the #653 work I'm doing now (which requires a change to foolscap: ​foolscap#105), I need to run a Tahoe trunk tree against a Foolscap trunk tree (to test changes I'm considering putting into the upcoming foolscap release). The normal way to do this (which used to work before we switched to setuptools) was:

PYTHONPATH=~/foolscap/trunk make test

I tried to do this yesterday, and here are the problems I ran into:

• the Foolscap .egg that was built in support/lib overrides anything else on PYTHONPATH, so I must delete the .egg first
• on OS-X, there's also an .egg put in the root of the tahoe tree (the #657 problem), so I must delete that one too
• "make test" takes forever (#591) and rebuilds a Foolscap .egg before running any tests, so I use "make quicktest" instead
• A Foolscap source tree (in my ~/foolscap/trunk directory) does not appear to qualify as fulfilling the setuptools dependency for foolscap[secure_connections]>=0.3.1. src/allmydata/scripts/runner.py has a line which does pkg_resources.require('allmydata-tahoe'), and that throws a DistributionNotFound exception. I commented that line out.
• the pkg_resources.load_entry_point-based support/bin/tahoe script does an implicit pkg_resources.require, also failing because it thinks it doesn't have Foolscap available.

I think the only way I can work around this is to use "setup.py develop TARGETDIR" on my Foolscap tree, and then set PYTHONPATH to point at TARGETDIR instead of ~/foolscap/trunk.

This is annoying. I really wish that PYTHONPATH could be used as it was originally intended, and that these eggs weren't getting in the way.

[warner]

fix formatting of Foolscap ticket link

[zooko]

The first two problems in your list of five problems are ​http://bugs.python.org/setuptools/issue53 (respect the PYTHONPATH). There have been two different patches posted to that ticket (the first was by me), and I don't currently understand the objection of the setuptools maintainer (PJE) to the most recent patch. Perhaps you could help by reading that ticket and figuring out what needs to change to get PJE to accept Mark Sienkiwicz's patch.

Alternately, we could continue our toothpick of setuptools (which I call zetuptoolz) by applying my patch or Mark Sienkiwicz's patch to zetuptoolz.

For what it is worth, this is going to be fixed one way or another in the long run, as either distutils core or various setuptools competitors move ahead and we will be able to switch to that in order to gain "respect the PYTHONPATH" without losing other functionality that we currently use.

Now, as for workarounds, if you build a .egg-info file, I think it would eliminate the last 3 of your 5 complaints. I'm not sure if that will do it. .egg-info files are created by default by modern Python distutils when building.

[warner]

Yup, by doing python setup.py build in the Foolscap tree, it created a foolscap.egg-info directory there, after which PYTHONPATH=~/foolscap/trunk make quicktest started working again (without commenting out the pkg_resources.require calls).

Also, PYTHONPATH=~/tahoe/trunk/support/lib/python2.5/site-packages python setup.py develop --prefix ~/tahoe/trunk/support put a foolscap.egg-link file in the tahoe lib directory, from which tahoe was able to find the foolscap code.

(I'm not used to doing python setup.py build with foolscap, because Foolscap doesn't need it for anything).

This works well enough for now. I look forward to seeing the "respect PYTHONPATH for .eggs" fix get into whatever packaging system we use. I guess I'll leave this ticket open until that happens. Thanks!

[warner]

I was mistaken. python setup.py build does not create a foolscap.egg-info directory. You must do a setup.py egg_info to get it to create the .egg-info . The setup.py develop merely creates it as a side-effect.

[cgalvan]

Note that once #668 is resolved, switching to Distribute, this issue may or may not be resolved as a result.

[zooko]

I opened a ticket with the Python Distribute project: ​http://bitbucket.org/tarek/distribute/issue/136/respect-the-pythonpath

[zooko]

zetuptoolz has a patch which is intended to fix this: ​http://tahoe-lafs.org/trac/zetuptoolz/changeset/20081115185932-92b7f-d451b10759897b0fa5d038f2c42cdcd3901289a8/trunk

If you wanted to try it out by installing zetuptoolz instead of your normal setuptools-or-distribute and try PYTHONPATH=~/foolscap/trunk make test, I would appreciate it.

[zooko]

Oh, it isn't just installing zetuptoolz -- this code gets written into your easy-install.pth. So to test this, read the patch and make the appropriate adjustment to the code in your easy-install.pth, and then see if PYTHONPATH=~/foolscap/trunk make test works. If so, then maybe we can extend zetuptoolz so that it hunts down all easy-install.pth's that it has write access to and fixes them. ;-)

Here is the code in question: ​zetuptoolz's easy_install.py

[zooko]

warner: are you willing to try this?

[zooko]

I've opened ​Launchpad bug #821000 to track the way this affects setuptools, distribute, Ubuntu, and Tahoe-LAFS (via our local fork of setuptools named "zetuptoolz").

[warner]

This is now resolved: by using a virtualenv, I can pip install --editable ~/my/foolscap to use a locally-modified Foolscap along with my locally-modified Tahoe. It's not $PYTHONPATH manipulation, but it works just as well.