incompatibility between zope.interface 3.6.3/3.6.4 and Nevow 0.10

[davidsarah]

The following error happens on python setup.py build when using Nevow 0.10 and zope.interface 3.6.4 (it appears to be the new version of zope.interface that introduces the problem, since we normally work fine with Nevow 0.10):

> python setup.py build
running darcsver
setup.py darcsver: using extant version file ['src/allmydata/_version.py']
running develop
Not found: tahoe-deps
Not found: ../tahoe-deps
running egg_info
writing requirements to src/allmydata_tahoe.egg-info/requires.txt
writing src/allmydata_tahoe.egg-info/PKG-INFO
writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt
writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt
writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt
writing requirements to src/allmydata_tahoe.egg-info/requires.txt
writing src/allmydata_tahoe.egg-info/PKG-INFO
writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt
writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt
writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt
reading manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
running build_ext
Processing setuptools_darcs-1.2.12.egg
removing '/home/tux/allmydata-tahoe-1.8.2/support/lib/python2.6/site-packages/setuptools_darcs-1.2.12.egg' (and everything under it)
Copying setuptools_darcs-1.2.12.egg to /home/tux/allmydata-tahoe-1.8.2/support/lib/python2.6/site-packages
setuptools-darcs 1.2.12 is already the active version in easy-install.pth

Installed /home/tux/allmydata-tahoe-1.8.2/support/lib/python2.6/site-packages/setuptools_darcs-1.2.12.egg
Creating /home/tux/allmydata-tahoe-1.8.2/support/lib/python2.6/site-packages/allmydata-tahoe.egg-link (link to src)
allmydata-tahoe 1.8.2 is already the active version in easy-install.pth
Installing tahoe script to support/bin

Installed /home/tux/allmydata-tahoe-1.8.2/src
Processing dependencies for allmydata-tahoe==1.8.2
Searching for Nevow>=0.6.0
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Best match: Nevow 0.10.0
Downloading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/Nevow-0.10.0.tar.bz2
Processing Nevow-0.10.0.tar.bz2
Running Nevow-0.10.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-moUdxw/Nevow-0.10.0/egg-dist-tmp-dXJymM
/tmp/easy_install-moUdxw/Nevow-0.10.0/formless/annotate.py:730: DeprecationWarning: object.__new__() takes no parameters
  rv = cls = InterfaceClass.__new__(cls, name, bases, dct)
Traceback (most recent call last):
  File "setup.py", line 417, in <module>
    **setup_args
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/develop.py", line 23, in run
    self.old_run()
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/develop.py", line 33, in old_run
    self.install_for_development()
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/develop.py", line 103, in install_for_development
    self.process_distribution(None, self.dist, not self.no_deps)
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/easy_install.py", line 526, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/pkg_resources.py", line 573, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/pkg_resources.py", line 823, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/pkg_resources.py", line 835, in obtain
    return installer(requirement)
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/easy_install.py", line 453, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/easy_install.py", line 483, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/easy_install.py", line 663, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/easy_install.py", line 938, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/command/easy_install.py", line 927, in run_setup
    run_setup(setup_script, args)
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/sandbox.py", line 62, in run_setup
    lambda: execfile(
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/sandbox.py", line 105, in run
    return func()
  File "/home/tux/allmydata-tahoe-1.8.2/setuptools-0.6c16dev3.egg/setuptools/sandbox.py", line 64, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 3, in <module>
    
  File "/tmp/easy_install-moUdxw/Nevow-0.10.0/nevow/__init__.py", line 143, in <module>
    
  File "/tmp/easy_install-moUdxw/Nevow-0.10.0/nevow/__init__.py", line 29, in load
    
  File "/tmp/easy_install-moUdxw/Nevow-0.10.0/nevow/util.py", line 183, in _namedAnyWithBuiltinTranslation
  File "/home/tux/allmydata-tahoe-1.8.2/Twisted-10.2.0-py2.6-linux-i686.egg/twisted/python/reflect.py", line 464, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/tux/allmydata-tahoe-1.8.2/Twisted-10.2.0-py2.6-linux-i686.egg/twisted/python/reflect.py", line 400, in _importAndCheckStack
    return __import__(importName)
  File "/tmp/easy_install-moUdxw/Nevow-0.10.0/formless/__init__.py", line 9, in <module>
    # See the README file for information on usage and redistribution.
  File "/tmp/easy_install-moUdxw/Nevow-0.10.0/formless/annotate.py", line 851, in <module>
  File "/tmp/easy_install-moUdxw/Nevow-0.10.0/formless/annotate.py", line 731, in __new__
  File "/home/tux/allmydata-tahoe-1.8.2/zope.interface-3.6.4-py2.6-linux-i686.egg/zope/interface/interface.py", line 685, in __hash__
    return hash((self.__name__, self.__module__))
AttributeError: 'MetaTypedInterface' object has no attribute '__name__'

Drew Perttula posted a workaround to ​monkey-patch zope.interface, but it is difficult for Tahoe to apply this workaround because the failure happens in setup.py build, before any run-time Tahoe code has executed.

[davidsarah]

attachment:dependency-updates.darcs.patch​ works by requiring one of the exact versions 3.3.1, 3.5.3 or 3.6.1 of zope.interface (both at build and run-time). These are the versions packaged in Debian, Ubuntu and pkgsrc. It also updates the dependency for Twisted as required by #1274 and #1429.

[davidsarah]

setup.py, _auto_deps.py, NEWS: dependency updates for Twisted and zope.interface. This version also removes the instruction to install pywin32 on Windows, which is no longer needed. fixes #1274, #1435. refs #1429

[davidsarah]

We still need upstream zope.interface to be fixed fairly soon, because:

• we want to be able to allow future versions;
• if an OS distribution packages the current version and a user has that package installed (i.e. as a single-version-externally-managed Python dist), setuptools will report a version conflict when building Tahoe-LAFS.

The sooner upstream is fixed, the less likely it is that a broken version will be packaged.

[davidsarah]

Update the dependency on zope.interface to fix an incompatiblity between Nevow and zope.interface 3.6.4. fixes #1435 (Patch bundle also contains the patch for #1438.)

[davidsarah]

Please review fix-1435.darcs.patch​ instead of dependency-updates.darcs.patch​. They do the same thing, but fix-1435.darcs.patch has the zope.interface and Twisted updates in separate darcs patches.

[zooko]

I reviewed attachment:fix-1435.darcs.patch​ and it looks good to me.

[david-sarah@…]

In 2ebaa916df66fd3f:

Update the dependency on zope.interface to fix an incompatiblity between Nevow and zope.interface 3.6.4. fixes #1435

[zooko]

The current fix is quite unsatisfying because the bug is really in Nevow, not in zope.interface, and the Nevow maintainers are working on a new release of Nevow that will fix it, but the zope.interface maintainers are not currently planning to release a new version of zope.interface that changes this.

If we ship Tahoe-LAFS v1.9 with the current constraint -- "zope.interface == 3.3.1, == 3.5.3, == 3.6.1", then once a new Nevow comes out user will be required to use an older zope.interface for no good reason when they could instead use a newer zope.interface and an newer Nevow.

[davidsarah]

This is no longer critical; a better specification of the dependency can (and must, I think) wait until the new version of Nevow has been released.

[zooko]

Per ​the launchpad ticket, Tres Seaver has gone ahead and released a new version of zope.interface, v3.6.5, which works-around this problem. So we could change the dependency to something like zope.interface < 3.6.4, > 3.6.4.

[davidsarah]

_auto_deps.py: relax the requirement for zope.interface. fixes #1435

[davidsarah]

Lenny has zope.interface 3.3.1: ​http://packages.debian.org/search?keywords=python-zopeinterface

relax-zope-interface-requirement.darcs.patch​ allows every release version of zope.interface since then according to ​http://pypi.python.org/pypi/zope.interface, except for 3.6.3 and 3.6.4 which result in the breakage with Nevow, and 3.6.5 which had an earlier version of the fix/workaround.

[zooko]

Why not zope.interface < 3.6.4, > 3.6.4? I just tried it in a little test project, the complete contents of which are this setup.py file:

from setuptools import setup

setup(name="testeee", install_requires=["zope.interface < 3.6.4, > 3.6.4"])

It worked as expected, whether zope.interface 3.6.1 was installed, zope.interface 3.6.4 was installed, or no zope.interface at all was installed. This was using distribute v0.6.15 (Ubuntu 11.04 Natty).

There are trade-offs between more missed alarms (in the extreme, allowing all versions except ones that we know don't work) and more false alarms (in the extreme, forbidding all versions except ones that we know work). Our current practice is to err much more toward the former extreme -- we don't like to exclude versions of dependencies unless we have good reason to think that they won't work. Documenting this and deciding whether to change it, is the topic of #1452. (gdt had some good suggestions on that ticket.)

So as far as I understand, to change our requirement on zope.interface in keeping with our current practice would be to set it to zope.interface > 3.6.4, > 3.6.4. The previous release had no constraint at all on which version of zope.interface: _auto_deps.py rev 5069 line 17

[zooko]

Oh, I overlooked that 3.6.3 is also incompatible, so I propose zope.interface < 3.6.3, > 3.6.4.

[davidsarah]

Replying to zooko:

So as far as I understand, to change our requirement on zope.interface in keeping with our current practice would be to set it to zope.interface < 3.6.4, > 3.6.4. The previous release had no constraint at all on which version of zope.interface: _auto_deps.py rev 5069 line 17

Well, that was almost certainly wrong. There must be some minimum version of zope.interface that we rely on; it's just that we don't know what it is.

I don't think we should have any requirements where the version is unconstrained. The lack of constraint on the versions of pyOpenSSL and mock is also almost certainly wrong.

[zooko]

Replying to davidsarah:

Well, that was almost certainly wrong. There must be some minimum version of zope.interface that we rely on; it's just that we don't know what it is.

Maybe so, but changing that is the subject of #1452. This ticket is about avoiding the incompatibility between zope.interface and Nevow.

[davidsarah]

Replying to zooko:

Replying to davidsarah:

Well, that was almost certainly wrong. There must be some minimum version of zope.interface that we rely on; it's just that we don't know what it is.

Maybe so, but changing that is the subject of #1452. This ticket is about avoiding the incompatibility between zope.interface and Nevow.

True, but there's little point in changing the requirement to < 3.6.3, > 3.6.4, and then changing it back to essentially what is in relax-zope-interface-requirement.darcs.patch​. If we're going to do the latter then we might as well do it now.

Note that there is no way to express a constraint "(version >= x and version < y) or version >= z" directly, other than by listing the versions from x to y.

[zooko]

Replying to davidsarah:

True, but there's little point in changing the requirement to < 3.6.3, > 3.6.4, and then changing it back to essentially what is in relax-zope-interface-requirement.darcs.patch​. If we're going to do the latter then we might as well do it now.

In my opinion we should not try to address #1452 for Tahoe-LAFS v1.9. I think we should commit a patch that fixes this ticket (working-around the incompatibility between zope.interface and Nevow), but which does not change our current policy of loose version requirements, and save that latter discussion for after v1.9.

Note that this is has immediate practical consequences--Brian was annoyed and delayed recently when working on the train because an earlier patch on this ticket had tightened the requirements of zope.interface beyond what it needed to. (Brian had an older, but perfectly functional, version of zope.interface installed.)

[davidsarah]

Running the test suite with zope.interface 3.6.5 results in these failures, due to a warning that needs to be suppressed:

[FAIL]: allmydata.test.test_runner.BinTahoe.test_version_no_noise

Traceback (most recent call last):
  File "/home/davidsarah/tahoe/snotty/src/allmydata/test/test_runner.py", line 207, in _cb
    and "from pkg_resources import load_entry_point" not in line)], str(res))
twisted.trial.unittest.FailTest: ('allmydata-tahoe: 1.8.2-r5156,\nfoolscap: 0.6.1,\npycryptopp: 0.5.25,\nzfec: 1.4.5,\nTwisted: 10.1.0,\nNevow: 0.10.0,\nzope.interface: unknown,\npython: 2.6.6,\nplatform: Linux-Ubuntu_10.10-x86_64-64bit_ELF,\npyOpenSSL: 0.10,\nsimplejson: 2.1.1,\npycrypto: 2.0.1,\npyasn1: unknown,\nmock: 0.8.0beta3,\nsqlite3: 2.4.1 [sqlite 3.7.2],\nsetuptools: 0.6c16dev3\n\n', "/home/davidsarah/tahoe/snotty/support/lib/python2.6/site-packages/zope.interface-3.6.5-py2.6-linux-x86_64.egg/zope/interface/interface.py:688: UserWarning: Hashing uninitialized InterfaceClass instance\n  warnings.warn('Hashing uninitialized InterfaceClass instance')\n", 0)
===============================================================================
[FAIL]: allmydata.test.test_runner.RunNode.test_client_no_noise

Traceback (most recent call last):
  File "/home/davidsarah/tahoe/snotty/src/allmydata/test/test_runner.py", line 511, in _cb2
    and "from pkg_resources import load_entry_point" not in line)], errstr)
twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/home/davidsarah/tahoe/snotty/support/lib/python2.6/site-packages/zope.interface-3.6.5-py2.6-linux-x86_64.egg/zope/interface/interface.py:688: UserWarning: Hashing uninitialized InterfaceClass instance
  warnings.warn('Hashing uninitialized InterfaceClass instance')
'
===============================================================================
[FAIL]: allmydata.test.test_system.SystemTest.test_filesystem_with_cli_in_subprocess

Traceback (most recent call last):
  File "/home/davidsarah/tahoe/snotty/src/allmydata/test/test_system.py", line 1761, in _check_succeeded
    self.failUnlessEqual(err, "")
twisted.trial.unittest.FailTest: not equal:
a = "/home/davidsarah/tahoe/snotty/support/lib/python2.6/site-packages/zope.interface-3.6.5-py2.6-linux-x86_64.egg/zope/interface/interface.py:688: UserWarning: Hashing uninitialized InterfaceClass instance\n  warnings.warn('Hashing uninitialized InterfaceClass instance')\n"
b = ''

[davidsarah]

_auto_deps.py: change the requirement for zope.interface to <= 3.6.2, >= 3.6.6. fixes #1435

[davidsarah]

zope.interface 3.6.6 doesn't fail due to the warning (I confirmed by grepping its source that 3.6.6 doesn't raise any warnings at all).

I'm fine with "<= 3.6.2, >= 3.6.6" for the Tahoe v1.9 release, although I still think we will end up changing the minimum zope.interface version to 3.3.1 (there was a bug that affects Python 2.5 support immediately before that, according to the changelog).

Note that the <= and >= constraints are correct because we don't know which post-3.6.2 version introduced the incompatibility, or which pre-3.6.6 version removed the UserWarning.

[davidsarah]

allmydata/init.py, test_version.py: make version parsing understand '<=', with test. refs #1435

[davidsarah]

relax-zope-interface-requirement-2.darcs.patch​ causes a spurious failure unless make-version-parsing-understand-le.darcs.patch​ is applied as well. (Actually, since zope.interface doesn't declare its version, we are unable to check it, but we need to be able to parse '<=' in a requirement in order to get to the point where we know that.)

[zooko]

+1 on attachment:make-version-parsing-understand-le.darcs.patch​ and attachment:relax-zope-interface-requirement-2.darcs.patch​.

[david-sarah@…]

In a48aee11b3ba7636:

(The changeset message doesn't reference this ticket)

[david-sarah@…]

In a98a4ec13f3353aa:

_auto_deps.py: change the requirement for zope.interface to <= 3.6.2, >= 3.6.6. fixes #1435

[david-sarah@…]

In [5189/ticket393-MDMF-2]:

allmydata/init.py, test_version.py: make version parsing understand '<=', with test. refs #1435

[david-sarah@…]

In [5190/ticket393-MDMF-2]:

_auto_deps.py: change the requirement for zope.interface to <= 3.6.2, >= 3.6.6. fixes #1435

[zooko]

That UserWarning is zope.interface warning about the poor behavior that Nevow does which caused this bug in the first place and that zope.interface currently works around. So I would expect that UserWarning to be present in newer version of zope.interface when used with Nevow 0.10.0.

Indeed, it fails with zope.interface 3.7.0:

allmydata.test.test_runner.RunNode.test_client_no_noise
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/zooko/playground/tahoe-lafs/393-2/src/allmydata/test/test_system.py", line 1766, in _check_succeeded
    self.failUnlessEqual(err, "")
twisted.trial.unittest.FailTest: not equal:
a = "/usr/local/lib/python2.7/dist-packages/zope.interface-3.7.0-py2.7-linux-x86_64.egg/zope/interface/interface.py:688: UserWarning: Hashing uninitialized InterfaceClass instance\n  warnings.warn('Hashing uninitialized InterfaceClass instance')\n"
b = ''

So, we should probably suppress that UserWarning and relax the requirement on zope.interface to <= 3.6.2, >= 3.6.5.

[zooko@…]

In f627af3b9fbe2975:

(The changeset message doesn't reference this ticket)

[davidsarah]

Cleanups for suppression of UserWarnings?. refs #1435

[zooko]

+1 on attachment:cleanup-userwarnings.darcs.patch​. I wonder why you didn't use darcs replace deprecation_imports warning_imports src/allmydata/__init__.py.

[david-sarah@…]

In c41bf1402944057b:

Cleanups for suppression of UserWarnings?. refs #1435