Twisted 16.0.0 causes test failure when importing cryptography.exceptions

[daira]

E.g. ​https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/116153170#L1350 :

allmydata.test.test_client ... Traceback (most recent call last):
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/trial/runner.py", line 511, in loadPackage
    module = modinfo.load()
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/modules.py", line 389, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/reflect.py", line 303, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/reflect.py", line 250, in _importAndCheckStack
    reraise(excValue, excTraceback)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/test/test_client.py", line 11, in <module>
    from allmydata.frontends.auth import NeedRootcapLookupScheme
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/frontends/auth.py", line 8, in <module>
    from twisted.conch.ssh import keys
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/conch/ssh/keys.py", line 16, in <module>
    from cryptography.exceptions import InvalidSignature
exceptions.ImportError: No module named cryptography.exceptions

[warner]

It looks like Twisted only declares a dependency on "cryptography" if you ask for the "conch" extra, as in pip install twisted[conch], or install_requires = [ "Twisted[conch]" ]. The previous twisted-15.5.0 had a [conch] dependency of "gmpy, pyasn1, pycrypto", which overlapped enough with what Tahoe declares dependencies on that things still worked. Twisted-16.0.0 changed conch to "gmpy, pyasn1, cryptography".

Tahoe has been in the practice of declaring a transitive closure of dependencies, and I think it's time we ended that and let pip have responsibility for figuring out the transitive set. (We can always add a requires.txt if we want to behave like an application, as opposed to a library, and also provide peep-style hashes of upstream tarballs/wheels). We should declare a dependency on Twisted[conch]. Foolscap should declare Twisted[tls] (added as ​foolscap#249).

And I think it might be appropriate for tahoe to have a windows-conditional dependency on Twisted[windows], since that's what triggers twisted to ask for pypiwin32.

[daira]

warner wrote:

Tahoe has been in the practice of declaring a transitive closure of dependencies, and I think it's time we ended that and let pip have responsibility for figuring out the transitive set.

+1

[warner]

Milestone renamed

[warner]

We'll bump the dependency to twisted[tls,conch] to cover both this issue, and to get Twisted's TLS stuff installed. (Ideally we'd leave that responsibility to Foolscap, but a setuptools bug means that "extras" only get honored in the first appearance of a package).

[warner]

Oops, the twisted[conch] triggered an inclusion of gmpy, which fails to build on most systems (because they lack the libgmp headers). It turns out that gmpy is pretty optional: the twisted code (specifically twisted/conch/ssh/common.py) uses it to monkey-patch the built-in pow() function with a faster version. (Maybe this is old? I thought python used libgmp for pow() already). If conch cannot import gmpy, it silently continues with the default pow().

So we're reducing the dependency to just twisted[tls], which currently gets us everything that twisted[conch] would get *except* for gmpy. And since gmpy is really optional, SFTP should still just work.

[Daira Hopwood <daira@…>]

In c3d54e0/trunk:

Back out the dependency on the conch extra for Twisted. refs ticket:2740

Signed-off-by: Daira Hopwood <daira@…>