python setup.py --help-commands raises exception

[zooko]

 zompu:~/playground/tahoe-lafs/pristine$ python setup.py --help-commands
Traceback (most recent call last):
  File "setup.py", line 417, in <module>
    **setup_args
  File "/usr/lib/python2.7/distutils/core.py", line 138, in setup
    ok = dist.parse_command_line()
  File "/home/zooko/playground/tahoe-lafs/pristine/setuptools-0.6c16dev3.egg/setuptools/dist.py", line 271, in parse_command_line
    result = _Distribution.parse_command_line(self)
  File "/usr/lib/python2.7/distutils/dist.py", line 464, in parse_command_line
    if self.handle_display_options(option_order):
  File "/usr/lib/python2.7/distutils/dist.py", line 669, in handle_display_options
    self.print_commands()
  File "/home/zooko/playground/tahoe-lafs/pristine/setuptools-0.6c16dev3.egg/setuptools/dist.py", line 403, in print_commands
    cmdclass = ep.load(False) # don't require extras, we're not running
  File "/home/zooko/playground/tahoe-lafs/pristine/setuptools-0.6c16dev3.egg/pkg_resources.py", line 1978, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
ImportError: No module named upload_docs

 zompu:~/playground/tahoe-lafs/pristine$ ./bin/tahoe --version
allmydata-tahoe: 1.8.2-r5043,
foolscap: 0.6.1,
pycryptopp: 0.5.28-r769,
zfec: 1.4.7,
Twisted: 10.1.0,
Nevow: 0.10.0,
zope.interface: unknown,
python: 2.7.1+,
platform: Linux-Ubuntu_11.04-x86_64-64bit_ELF,
pyOpenSSL: 0.10,
simplejson: 2.1.2,
pycrypto: 2.1.0,
pyasn1: unknown,
mock: 0.7.1,
sqlite3: 2.6.0 [sqlite 3.7.4],
setuptools: 0.6c16dev3

[davidsarah]

embedded zetuptoolz: tolerate exceptions when loading extra commands for "setup.py --help-commands".

[davidsarah]

tolerate-exceptions-in-help-commands.darcs.patch​ causes exceptions when loading "extra" commands to be ignored for the purpose of compiling the --help-commands list (that is, the broken command will not be listed in the help).

[zooko]

Does anyone know why the "upload_docs" command is broken? I don't know where it is even defined or how it got into my (Ubuntu Natty) system. It isn't defined in the tahoe-lafs codebase or the zetuptoolz codebase.

We should presumably report a bug to its author...

Otherwise, the attachment:tolerate-exceptions-in-help-commands.darcs.patch​ seems pretty good. Should we catch more specifically ImportError instead of all exceptions for now?

[davidsarah]

Replying to zooko:

Does anyone know why the "upload_docs" command is broken? I don't know where it is even defined or how it got into my (Ubuntu Natty) system. It isn't defined in the tahoe-lafs codebase or the zetuptoolz codebase.

That's the trouble with loading extensions from random code on the sys.path. (I'm not even going to look at how setuptools decides what to load; it would probably upset me.)

We should presumably report a bug to its author...

It appears to be part of setuptools and/or distribute. I can't find any documentation for it in setuptools, but the documentation for the corresponding command in Distutils2 is ​here. So the author is probably either PJE or Tarek.

Otherwise, the attachment:tolerate-exceptions-in-help-commands.darcs.patch​ seems pretty good. Should we catch more specifically ImportError instead of all exceptions for now?

I don't know that loading a broken command would necessarily throw ImportError. The wider catch seems fine to me for --help-commands; note that actually trying to use the command would throw the original exception.

[zooko]

I did some experimenting:

1. If I sudo apt-get remove python-setuptools, thus removing Distribute v0.6.15, which is ​distributed by Ubuntu in Natty 11.04 under the name python-setuptools, then the exception ImportError: No module named upload_docs stops happening. If I sudo apt-get install python-setuptools to put it back on my system, then the exception starts happening again.

2. If (while Ubuntu Natty's python-setuptools package is installed), I comment out ​line 72 and 73 of setup.py, which injects zetuptoolz into the current Python process, then the exception stops happening. If I comment those lines back in, the exception starts happening again.

So I guess this is not a bug in Distribute itself, nor in zetuptoolz either, but is a conflict between the two of them. I guess that Distribute has added upload_docs to a list of known commands, but since zetuptoolz instead of Distribute is loaded into this Python process, upload_docs can't be imported. I'm guessing that upload_docs isn't in zetuptoolz but is in Distribute.

Even if this is the case, then the solution encoded into attachment:tolerate-exceptions-in-help-commands.darcs.patch​ is still potentially a good solution. If there is a command listed in the available commands, but attempting to import it to get its usage string raises ImportError, then perhaps just skipping it and not listing it in the --help-commands output is fine.

Is there a useful unit test we could write to detect future regressions in attachment:tolerate-exceptions-in-help-commands.darcs.patch​ or conflicts between it and other code?

Also: this makes me more interested in catching narrowly ImportError, since I currently think that this particular kind of conflict should not result in any other error than that.

[davidsarah]

thedod was confused by this bug. We should probably just apply the patch (and bump the zetuptoolz version).

[davidsarah]

Fixed for the embedded zetuptoolz egg in ​0e20ab60 and ​5ab27f58. For the corresponding patch to upstream zetuptools see ​https://tahoe-lafs.org/trac/zetuptoolz/ticket/4.

I did narrow the catch to ImportError as suggested in comment:4.