Tahoe 1.8.2b1 does not locate locally installed setuptools

[killyourtv]

Debian Squeeze (testing), x64.

Software versions:

foolscap: 0.6.1,
pycryptopp: 0.5.25,
zfec: 1.4.9,
Twisted: 10.1.0,
Nevow: 0.10.0,
zope.interface: 3.5.3,
python: 2.6.6,
pyOpenSSL: 0.10,
simplejson: 2.1.1,
pycrypto: 2.1.0,
pyasn1: 0.0.11a,
mock: 0.6.0,
sqlite3: 2.4.1,
sqlite: 3.7.4,
setuptools: 0.6.14

$ tahoe start STARTING '/home/$USERNAME/.tahoe'

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/twisted/application/app.py", line 626, in run
    runApp(config)
  File "/usr/lib/python2.6/dist-packages/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/usr/lib/python2.6/dist-packages/twisted/application/app.py", line 374, in run
    self.application = self.createOrGetApplication()
  File "/usr/lib/python2.6/dist-packages/twisted/application/app.py", line 439, in createOrGetApplication
    application = getApplication(self.config, passphrase)
--- <exception caught here> ---
  File "/usr/lib/python2.6/dist-packages/twisted/application/app.py", line 450, in getApplication
    application = service.loadApplication(filename, style, passphrase)
  File "/usr/lib/python2.6/dist-packages/twisted/application/service.py", line 390, in loadApplication
    application = sob.loadValueFromFile(filename, 'application', passphrase)
  File "/usr/lib/python2.6/dist-packages/twisted/persisted/sob.py", line 210, in loadValueFromFile
    exec fileObj in d, d
  File "tahoe-client.tac", line 10, in <module>
    c = client.Client()
  File "/usr/lib/pymodules/python2.6/allmydata/client.py", line 133, in __init__
    node.Node.__init__(self, basedir)
  File "/usr/lib/pymodules/python2.6/allmydata/node.py", line 75, in __init__
    self.log("Node constructed. " + get_package_versions_string())
  File "/usr/lib/pymodules/python2.6/allmydata/__init__.py", line 374, in get_package_versions_string
    errors = cross_check_pkg_resources_versus_import()
  File "/usr/lib/pymodules/python2.6/allmydata/__init__.py", line 264, in cross_check_pkg_resources_versus_import
    pr_ver, pr_loc = pkg_resources_vers_and_locs[name]
exceptions.KeyError: 'setuptools'

Failed to load application: 'setuptools'

But setuptools *is* installed.

$ LANG=en apt-cache policy python-setuptools
python-setuptools:
  Installed: 0.6.14-4
  Candidate: 0.6.14-4

Tahoe 1.8.1 was able to use the locally installed version. Additionally, Python is able to import the module.

>>> import setuptools
import setuptools # directory /usr/lib/python2.6/dist-packages/setuptools
# /usr/lib/python2.6/dist-packages/setuptools/__init__.pyc matches /usr/lib/python2.6/dist-packages/setuptools/__init__.py
import setuptools # precompiled from /usr/lib/python2.6/dist-packages/setuptools/__init__.pyc
# /usr/lib/python2.6/dist-packages/setuptools/extension.pyc matches /usr/lib/python2.6/dist-packages/setuptools/extension.py
import setuptools.extension # precompiled from /usr/lib/python2.6/dist-packages/setuptools/extension.pyc
import distutils # directory /usr/lib/python2.6/distutils
# /usr/lib/python2.6/distutils/__init__.pyc matches /usr/lib/python2.6/distutils/__init__.py
import distutils # precompiled from /usr/lib/python2.6/distutils/__init__.pyc
# /usr/lib/python2.6/distutils/core.pyc matches /usr/lib/python2.6/distutils/core.py
import distutils.core # precompiled from /usr/lib/python2.6/distutils/core.pyc
# /usr/lib/python2.6/distutils/debug.pyc matches /usr/lib/python2.6/distutils/debug.py
import distutils.debug # precompiled from /usr/lib/python2.6/distutils/debug.pyc
# /usr/lib/python2.6/distutils/errors.pyc matches /usr/lib/python2.6/distutils/errors.py
import distutils.errors # precompiled from /usr/lib/python2.6/distutils/errors.pyc
# /usr/lib/python2.6/distutils/util.pyc matches /usr/lib/python2.6/distutils/util.py
import distutils.util # precompiled from /usr/lib/python2.6/distutils/util.pyc
# /usr/lib/python2.6/string.pyc matches /usr/lib/python2.6/string.py
import string # precompiled from /usr/lib/python2.6/string.pyc
# /usr/lib/python2.6/re.pyc matches /usr/lib/python2.6/re.py
import re # precompiled from /usr/lib/python2.6/re.pyc
# /usr/lib/python2.6/sre_compile.pyc matches /usr/lib/python2.6/sre_compile.py
import sre_compile # precompiled from /usr/lib/python2.6/sre_compile.pyc
import _sre # builtin
# /usr/lib/python2.6/sre_parse.pyc matches /usr/lib/python2.6/sre_parse.py
import sre_parse # precompiled from /usr/lib/python2.6/sre_parse.pyc
# /usr/lib/python2.6/sre_constants.pyc matches /usr/lib/python2.6/sre_constants.py
import sre_constants # precompiled from /usr/lib/python2.6/sre_constants.pyc
import strop # builtin
# /usr/lib/python2.6/distutils/dep_util.pyc matches /usr/lib/python2.6/distutils/dep_util.py
import distutils.dep_util # precompiled from /usr/lib/python2.6/distutils/dep_util.pyc
# /usr/lib/python2.6/distutils/spawn.pyc matches /usr/lib/python2.6/distutils/spawn.py
import distutils.spawn # precompiled from /usr/lib/python2.6/distutils/spawn.pyc
# /usr/lib/python2.6/distutils/log.pyc matches /usr/lib/python2.6/distutils/log.py
import distutils.log # precompiled from /usr/lib/python2.6/distutils/log.pyc
# /usr/lib/python2.6/distutils/dist.pyc matches /usr/lib/python2.6/distutils/dist.py
import distutils.dist # precompiled from /usr/lib/python2.6/distutils/dist.pyc
# /usr/lib/python2.6/copy.pyc matches /usr/lib/python2.6/copy.py
import copy # precompiled from /usr/lib/python2.6/copy.pyc
# /usr/lib/python2.6/distutils/fancy_getopt.pyc matches /usr/lib/python2.6/distutils/fancy_getopt.py
import distutils.fancy_getopt # precompiled from /usr/lib/python2.6/distutils/fancy_getopt.pyc
# /usr/lib/python2.6/getopt.pyc matches /usr/lib/python2.6/getopt.py
import getopt # precompiled from /usr/lib/python2.6/getopt.pyc
# /usr/lib/python2.6/distutils/cmd.pyc matches /usr/lib/python2.6/distutils/cmd.py
import distutils.cmd # precompiled from /usr/lib/python2.6/distutils/cmd.pyc
# /usr/lib/python2.6/distutils/dir_util.pyc matches /usr/lib/python2.6/distutils/dir_util.py
import distutils.dir_util # precompiled from /usr/lib/python2.6/distutils/dir_util.pyc
# /usr/lib/python2.6/distutils/file_util.pyc matches /usr/lib/python2.6/distutils/file_util.py
import distutils.file_util # precompiled from /usr/lib/python2.6/distutils/file_util.pyc
# /usr/lib/python2.6/distutils/archive_util.pyc matches /usr/lib/python2.6/distutils/archive_util.py
import distutils.archive_util # precompiled from /usr/lib/python2.6/distutils/archive_util.pyc
# /usr/lib/python2.6/distutils/config.pyc matches /usr/lib/python2.6/distutils/config.py
import distutils.config # precompiled from /usr/lib/python2.6/distutils/config.pyc
# /usr/lib/python2.6/ConfigParser.pyc matches /usr/lib/python2.6/ConfigParser.py
import ConfigParser # precompiled from /usr/lib/python2.6/ConfigParser.pyc
# /usr/lib/python2.6/distutils/extension.pyc matches /usr/lib/python2.6/distutils/extension.py
import distutils.extension # precompiled from /usr/lib/python2.6/distutils/extension.pyc
# /usr/lib/python2.6/dist-packages/setuptools/dist.pyc matches /usr/lib/python2.6/dist-packages/setuptools/dist.py
import setuptools.dist # precompiled from /usr/lib/python2.6/dist-packages/setuptools/dist.pyc
# /usr/lib/python2.6/dist-packages/setuptools/depends.pyc matches /usr/lib/python2.6/dist-packages/setuptools/depends.py
import setuptools.depends # precompiled from /usr/lib/python2.6/dist-packages/setuptools/depends.pyc
# /usr/lib/python2.6/__future__.pyc matches /usr/lib/python2.6/__future__.py
import __future__ # precompiled from /usr/lib/python2.6/__future__.pyc
import imp # builtin
import marshal # builtin
# /usr/lib/python2.6/distutils/version.pyc matches /usr/lib/python2.6/distutils/version.py
import distutils.version # precompiled from /usr/lib/python2.6/distutils/version.pyc
import setuptools.command # directory /usr/lib/python2.6/dist-packages/setuptools/command
# /usr/lib/python2.6/dist-packages/setuptools/command/__init__.pyc matches /usr/lib/python2.6/dist-packages/setuptools/command/__init__.py
import setuptools.command # precompiled from /usr/lib/python2.6/dist-packages/setuptools/command/__init__.pyc
# /usr/lib/python2.6/dist-packages/setuptools/command/install_scripts.pyc matches /usr/lib/python2.6/dist-packages/setuptools/command/install_scripts.py
import setuptools.command.install_scripts # precompiled from /usr/lib/python2.6/dist-packages/setuptools/command/install_scripts.pyc
import distutils.command # directory /usr/lib/python2.6/distutils/command
# /usr/lib/python2.6/distutils/command/__init__.pyc matches /usr/lib/python2.6/distutils/command/__init__.py
import distutils.command # precompiled from /usr/lib/python2.6/distutils/command/__init__.pyc
# /usr/lib/python2.6/distutils/command/install_scripts.pyc matches /usr/lib/python2.6/distutils/command/install_scripts.py
import distutils.command.install_scripts # precompiled from /usr/lib/python2.6/distutils/command/install_scripts.pyc
# /usr/lib/python2.6/dist-packages/pkg_resources.pyc matches /usr/lib/python2.6/dist-packages/pkg_resources.py
import pkg_resources # precompiled from /usr/lib/python2.6/dist-packages/pkg_resources.pyc
import time # builtin
# /usr/lib/python2.6/urlparse.pyc matches /usr/lib/python2.6/urlparse.py
import urlparse # precompiled from /usr/lib/python2.6/urlparse.pyc
# /usr/lib/python2.6/collections.pyc matches /usr/lib/python2.6/collections.py
import collections # precompiled from /usr/lib/python2.6/collections.pyc
import _collections # builtin
import operator # builtin
# /usr/lib/python2.6/keyword.pyc matches /usr/lib/python2.6/keyword.py
import keyword # precompiled from /usr/lib/python2.6/keyword.pyc
# /usr/lib/python2.6/pkgutil.pyc matches /usr/lib/python2.6/pkgutil.py
import pkgutil # precompiled from /usr/lib/python2.6/pkgutil.pyc
import zope # directory /usr/lib/python2.6/dist-packages/zope
# /usr/lib/python2.6/dist-packages/zope/__init__.pyc matches /usr/lib/python2.6/dist-packages/zope/__init__.py
import zope # precompiled from /usr/lib/python2.6/dist-packages/zope/__init__.pyc
# /usr/lib/python2.6/distutils/command/bdist.pyc matches /usr/lib/python2.6/distutils/command/bdist.py
import distutils.command.bdist # precompiled from /usr/lib/python2.6/distutils/command/bdist.pyc
# /usr/lib/python2.6/dist-packages/setuptools/command/install.pyc matches /usr/lib/python2.6/dist-packages/setuptools/command/install.py
import setuptools.command.install # precompiled from /usr/lib/python2.6/dist-packages/setuptools/command/install.pyc
# /usr/lib/python2.6/glob.pyc matches /usr/lib/python2.6/glob.py
import glob # precompiled from /usr/lib/python2.6/glob.pyc
# /usr/lib/python2.6/fnmatch.pyc matches /usr/lib/python2.6/fnmatch.py
import fnmatch # precompiled from /usr/lib/python2.6/fnmatch.pyc
# /usr/lib/python2.6/distutils/command/install.pyc matches /usr/lib/python2.6/distutils/command/install.py
import distutils.command.install # precompiled from /usr/lib/python2.6/distutils/command/install.pyc
# /usr/lib/python2.6/distutils/sysconfig.pyc matches /usr/lib/python2.6/distutils/sysconfig.py
import distutils.sysconfig # precompiled from /usr/lib/python2.6/distutils/sysconfig.pyc
# /usr/lib/python2.6/dist-packages/setuptools/command/sdist.pyc matches /usr/lib/python2.6/dist-packages/setuptools/command/sdist.py
import setuptools.command.sdist # precompiled from /usr/lib/python2.6/dist-packages/setuptools/command/sdist.pyc
# /usr/lib/python2.6/distutils/command/sdist.pyc matches /usr/lib/python2.6/distutils/command/sdist.py
import distutils.command.sdist # precompiled from /usr/lib/python2.6/distutils/command/sdist.pyc
# /usr/lib/python2.6/distutils/text_file.pyc matches /usr/lib/python2.6/distutils/text_file.py
import distutils.text_file # precompiled from /usr/lib/python2.6/distutils/text_file.pyc
# /usr/lib/python2.6/distutils/filelist.pyc matches /usr/lib/python2.6/distutils/filelist.py
import distutils.filelist # precompiled from /usr/lib/python2.6/distutils/filelist.pyc
# /usr/lib/python2.6/dist-packages/setuptools/command/install_lib.pyc matches /usr/lib/python2.6/dist-packages/setuptools/command/install_lib.py
import setuptools.command.install_lib # precompiled from /usr/lib/python2.6/dist-packages/setuptools/command/install_lib.pyc
# /usr/lib/python2.6/distutils/command/install_lib.pyc matches /usr/lib/python2.6/distutils/command/install_lib.py
import distutils.command.install_lib # precompiled from /usr/lib/python2.6/distutils/command/install_lib.pyc

[zooko]

Thank you for the bug report! Could you also please post the results of python -c 'import pkg_resources;print pkg_resources.require("setuptools")'?

It looks like the behavior of stopping with an uncaught KeyError exception is due to a simple mistake -- would you be so kind as to apply the following patch and try again? If that patch works then it should emit a specific warning about the setuptools dependency but otherwise proceed to start-up. Thanks!

diff -rN -u old-trunk/src/allmydata/__init__.py new-trunk/src/allmydata/__init__.py
--- old-trunk/src/allmydata/__init__.py 2011-01-27 11:32:16.000000000 -0700
+++ new-trunk/src/allmydata/__init__.py 2011-01-27 11:32:22.000000000 -0700
@@ -260,6 +260,7 @@
             if name not in pkg_resources_vers_and_locs:
                 errors.append("Warning: dependency %s (version %s imported from %r) was not found by pkg_resources."
                               % (name, imp_ver, imp_loc))
+                continue
 
             pr_ver, pr_loc = pkg_resources_vers_and_locs[name]
             try:

(I'll also attach it as attachment:handle-missing-dep.patch.txt​.)

[killyourtv]

Thank you for the quick reply. :)

$ python -c 'import pkg_resources;print pkg_resources.require("setuptools")'
[distribute 0.6.14 (/usr/lib/python2.6/dist-packages)]

After applying the patch, tahoe 1.8.2b1 starts successfully and connects to the introducer on the I2P network.

If anything else would be helpful from my end, please let me know.

$ tahoe --version
allmydata-tahoe: 1.8.2b1,
foolscap: 0.6.1,
pycryptopp: 0.5.25,
zfec: 1.4.9,
Twisted: 10.1.0,
Nevow: 0.10.0,
zope.interface: unknown,
python: 2.6.6,
platform: Linux-debian_6.0-x86_64-64bit_ELF,
pyOpenSSL: 0.10,
simplejson: 2.1.1,
pycrypto: 2.1.0,
pyasn1: unknown,
mock: 0.6.0,
sqlite3: 2.4.1,
sqlite: 3.7.4,
setuptools: 0.6

Warning: dependency setuptools (version 0.6 imported from '/usr/lib/python2.6/dist-packages') was not found by pkg_resources.
Warning: dependency distribute (version 0.6.14) found by pkg_resources not found by import.

For debugging purposes, the PYTHONPATH was
  None
install_requires was
  ['setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface', 'Twisted >= 2.4.0', 'foolscap[secure_connections] >= 0.6.1', 'Nevow >= 0.6.0', 'pycrypto == 2.0.1, == 2.1.0, >= 2.3', 'pyasn1 >= 0.0.8a', 'mock', 'pycryptopp >= 0.5.20']
sys.path after importing pkg_resources was
  /usr/lib/pymodules/python2.6:
  /usr/lib/python2.6/dist-packages:
  /usr/bin:
  /usr/lib/python2.6:
  /usr/lib/python2.6/plat-linux2:
  /usr/lib/python2.6/lib-tk:
  /usr/lib/python2.6/lib-old:
  /usr/lib/python2.6/lib-dynload:
  /usr/local/lib/python2.6/dist-packages:
  /usr/lib/python2.6/dist-packages/PIL:
  /usr/lib/python2.6/dist-packages/gst-0.10:
  /usr/lib/pymodules/python2.6/gtk-2.0:
  /usr/lib/python2.6/dist-packages/wx-2.8-gtk2-unicode

[davidsarah]

I reviewed attachment:handle-missing-dep.patch.txt​ and it looks correct.

[davidsarah]

Replying to killyourtv:

After applying the patch, tahoe 1.8.2b1 starts successfully and connects to the introducer on the I2P network. [...]

$ tahoe --version
allmydata-tahoe: 1.8.2b1,
[...]
setuptools: 0.6

Warning: dependency setuptools (version 0.6 imported from
'/usr/lib/python2.6/dist-packages') was not found by pkg_resources.
Warning: dependency distribute (version 0.6.14) found by pkg_resources
not found by import.

For debugging purposes, the PYTHONPATH was [...]

With the patch, this is now behaving exactly as designed. However, the specific case of "import setuptools" giving distribute faked to look like setuptools 0.6, is common enough (and reasonably harmless enough) that we should perhaps special-case it to suppress the warning. It would also be nice if the -version[-and-path] output were to report that the imported setuptools is actually distribute.

[davidsarah]

Note that attachment:fix-1339.darcs.patch​ would be instead of handle-missing-dep.patch.txt .

[davidsarah]

src/allmydata/init.py: fix #1339, give an indication in the --version[-and-path] output of when the imported setuptools is distribute, and use a separate element in _vers_and_locs_list tuples for information other than the package name and location. This also changes slightly how the sqlite version is reported. [Use this version of the patch rather than the previous version, which was missing a line of code.]

[davidsarah]

src/allmydata/test/test_version.py: update to take into account change from pairs to triples in versions-and-locations list. refs #1339

[david-sarah@…]

In c71a19b9ed3bafe9:

src/allmydata/init.py: fix #1339, give an indication in the --version[-and-path] output of when the imported setuptools is distribute, and use a separate element in _vers_and_locs_list tuples for information other than the package name and location. This also changes slightly how the sqlite version is reported.

[zooko]

reviewed by me

[zooko]

In a new ticket #1341, killyourtv wrote:

Sorry, I'm back again.

After applying the patch in changeset:4994, the tests fail with the following output:

===============================================================================
[ERROR]: allmydata.test.test_version.CheckRequirement.test_check_requirement

Traceback (most recent call last):
  File "/build/$USERNAME-tahoe-lafs_1.8.2~beta1-2-amd64-ZHbyMn/tahoe-lafs-1.8.2~beta1/src/allmydata/test/test_version.py", line 12, in test_check_requirement
    check_requirement("setuptools >= 0.6c6", {"setuptools": ("0.6", "")})
  File "/build/$USERNAME-tahoe-lafs_1.8.2~beta1-2-amd64-ZHbyMn/tahoe-lafs-1.8.2~beta1/src/allmydata/__init__.py", line 214, in check_requirement
    (actual, location, comment) = vers_and_locs[name]
exceptions.ValueError: need more than 2 values to unpack
-------------------------------------------------------------------------------
Ran 839 tests in 521.128s

FAILED (skips=17, expectedFailures=3, errors=1, successes=818)

[zooko]

This is my fault -- sorry! I didn't run the tests after reviewing c71a19b9ed3bafe9 and before committing it to trunk, and I didn't notice, in my review, that there were tests which were testing the previous behavior of this code (which manipulates a name, location, and version but not a name, location, version, and comment).

(Also I forgot that no changes at all were supposed to be applied to trunk without the Release Manager's approval.)

I will push a patch to trunk to fix the tests.

[zooko@…]

In 6f84f1b869baa9f8:

setup: adjust tests to use the new interface of check_requirement which has a 3-tuples instead of a 2-tuple
fixes #1339

[killyourtv]

Just to follow up:

$ tahoe --version
allmydata-tahoe: 1.8.2b1,
foolscap: 0.6.1,
pycryptopp: 0.5.25,
zfec: 1.4.9,
Twisted: 10.1.0,
Nevow: 0.10.0,
zope.interface: unknown,
python: 2.6.6,
platform: Linux-debian_6.0-x86_64-64bit_ELF,
pyOpenSSL: 0.10,
simplejson: 2.1.1,
pycrypto: 2.1.0,
pyasn1: unknown,
mock: 0.6.0,
sqlite3: 2.4.1 [sqlite 3.7.4],
setuptools: 0.6 [distribute]

Thanks for the help. :)

[davidsarah]

Sorry, I should have combined attachment:fix-to-test.darcs.patch​ into the other patch, to make it clearer that they both needed to be committed.

The 6f84f1b869baa9f8 changeset that zooko pushed to fix this isn't quite right because the comment field would normally be None rather than "" (although check_requirement doesn't look at that field other than to report modules that couldn't be imported).

[davidsarah]

Change tests for check_requirement to ensure that it accepts None in the comment field of a (version, location, comment) triple, and add a test that it raises ImportError? when a module couldn't be imported. Also fix typo in a comment. refs #1339

[david-sarah@…]

In d318122cd25591cb:

(The changeset message doesn't reference this ticket)