tahoe import machinery hides details of import errors

[zooko]

arc had a hard time getting Tahoe-LAFS working. It turned out that the problem was binary extension modules from a different operating system were in his PYTHONPATH. But in order to learn that he had to import that package directly -- the Tahoe-LAFS import machinery hid that useful diagnostic information about why the import failed and instead just told him "zfec couldn't be imported".

cd ~/dw ;PYTHONPATH=src:support/lib/python2.6/site-packages:/home/arc/tpsrc/pycryptopp-0.5.29:/usr/lib/pymodules/python2.6 python -c 'import zfec ; print zfec'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/arc/dw/support/lib/python2.6/site-packages/zfec-1.4.10-py2.6-macosx-10.6-universal.egg/zfec/__init__.py", line 18, in <module>
    from _fec import Encoder, Decoder, Error
ImportError: /home/arc/dw/support/lib/python2.6/site-packages/zfec-1.4.10-py2.6-macosx-10.6-universal.egg/zfec/_fec.so: invalid ELF header

cd ~/dw ;PYTHONPATH=src:support/lib/python2.6/site-packages:/home/arc/tpsrc/pycryptopp-0.5.29:/usr/lib/pymodules/python2.6 python -c 'import allmydata ; print allmydata'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/arc/dw/src/allmydata/__init__.py", line 375, in <module>
    check_all_requirements()
  File "/home/arc/dw/src/allmydata/__init__.py", line 373, in check_all_requirements
    raise PackagingError(get_error_string(errors, debug=True))
allmydata.PackagingError:
ImportError: could not import 'zfec' for requirement 'zfec >= 1.1.0'
ImportError: could not import 'pycryptopp' for requirement 'pycryptopp >= 0.5.20'
 
For debugging purposes, the PYTHONPATH was
  'src:support/lib/python2.6/site-packages:/home/arc/tpsrc/pycryptopp-0.5.29:/usr/lib/pymodules/python2.6'
install_requires was
  ['setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface', 'Twisted >= 2.4.0', 'foolscap[secure_connections] >= 0.6.1', 'Nevow >= 0.6.0', 'pycrypto == 2.0.1, == 2.1.0, >= 2.3', 'pyasn1 >= 0.0.8a', 'mock', 'pycryptopp >= 0.5.20']
sys.path after importing pkg_resources was
  :
  /home/arc/dw/support/lib/python2.6/site-packages/setuptools_darcs-1.2.12.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/pycryptopp-0.5.29-py2.6-macosx-10.6-universal.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/mock-0.7.0-py2.6.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/pyasn1-0.0.13a-py2.6.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/pycrypto-2.3-py2.6-macosx-10.6-universal.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/Nevow-0.10.0-py2.6.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/foolscap-0.6.1-py2.6.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/simplejson-2.1.3-py2.6-macosx-10.6-universal.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/zfec-1.4.10-py2.6-macosx-10.6-universal.egg:
  /home/arc/dw/support/lib/python2.6/site-packages/setuptools-0.6c16dev3.egg:
  /usr/local/lib/python2.6/dist-packages/coverage-3.4-py2.6-linux-x86_64.egg:
  /usr/local/lib/python2.6/dist-packages/pyutil-1.8.4-py2.6.egg:
  /usr/local/lib/python2.6/dist-packages/argparse-1.1-py2.6.egg:
  /usr/local/lib/python2.6/dist-packages/zbase32-1.1.3-py2.6.egg:
  /home/arc/dw/src:
  /home/arc/dw/support/lib/python2.6/site-packages:
  /home/arc/tpsrc/pycryptopp-0.5.29:
  /usr/lib/pymodules/python2.6:
  /usr/lib/python2.6:
  /usr/lib/python2.6/plat-linux2:
  /usr/lib/python2.6/lib-tk:
  /usr/lib/python2.6/lib-old:
  /usr/lib/python2.6/lib-dynload:
  /usr/lib/python2.6/dist-packages:
  /usr/lib/python2.6/dist-packages/PIL:
  /usr/lib/python2.6/dist-packages/gst-0.10:
  /usr/lib/python2.6/dist-packages/gtk-2.0:
  /usr/lib/pymodules/python2.6/gtk-2.0:
  /usr/lib/python2.6/dist-packages/wx-2.8-gtk2-unicode:
  /usr/local/lib/python2.6/dist-packages

[davidsarah]

allmydata/init.py: preserve the message of ImportErrors? in the package versions string. fixes #1389

[davidsarah]

This is a bit tricky to test because 'allmydata' has already been imported in the process that is running trial. We'd have to arrange for an import to fail in a subprocess. Does it need a test? (I've checked manually that it works, by hacking foolscap to raise an ImportError.)

[davidsarah]

The traceback of the original ImportError (most usefully the filename and line number where the exception occurred) is still lost. Do we need that, or will the message usually be sufficient?

[zooko]

The patch looks good to me -- +1. Please review the attached test.

[zooko]

Please review attachment:1389-include-source-frame.darcs.patch​ which includes information about where the ImportError? originated. Includes test.

[davidsarah]

+1 on both these patches, with suggestions:

• use mock to patch __builtins__['__import__']
• traceback.extract_tb(etrace)[-1] returns a tuple rather than a string, so the comment element won't have the same type as it has in other cases. "str(traceback.extract_tb(etrace)[-1])" would work (in which case the test only needs "'wheeeyo' in stuff[2]").
• update "# comment is the module name and message of the original ImportError" to something like "# comment is the stringified traceback entry for the original ImportError"

[zooko]

I like to preserve structure and squash to string as late as possible, in general.

[davidsarah]

OK, I've checked that nothing else assumes the comment is a string, so we can leave it as a tuple. I will change the comment to "# comment is the last tuple in the traceback for the original ImportError" and commit to trunk.

[zooko]

Thanks! :-)

[davidsarah]

I haven't committed yet because I noticed that we're now missing the actual message of the exception. The test succeeds because the source line that raises the exception includes the marker string "wheeeyo".

[davidsarah]

packaging: show also the message of the original ImportError?. Ensure that an empty traceback is tolerated. Updates and makes minor cleanups to test. refs #1389

[davidsarah]

refactored version of the same patches

[zooko]

+1 on attachment:1389-refactored.patch​

[david-sarah@…]

In 449b7ad8c7889d82:

allmydata/init.py: preserve the message and last traceback entry (file, line number, function, and source line) of ImportErrors? in the package versions string. fixes #1389

[david-sarah@…]

In a09d7300e022dd51:

tests: add test for including the ImportError? message and traceback entry in the summary of errors from importing dependencies. refs #1389

[david-sarah@…]

In fffb3ae75b8225a9:

Fix a test failure in test_package_initialization on Python 2.4.x due to exceptions being stringified differently than in later versions of Python. refs #1389