`pip` managed install is broken.

[nejucomo]

Synopsis:

The command pip install allmydata-tahoe fails because of a bug in the Nevow dependency.

Known workaround:

Install twisted first with pip. This works for me:

$ pip install twisted && pip install allmydata-tahoe

Details:

It appears related to Nevow. I'm hopping on #twisted to ask for advice.

$ mkdir ~/tmp/tahoe-install-test

$ cd !$
cd ~/tmp/tahoe-install-test

$ virtualenv .
Using real prefix '/usr/local/bin/../stow/Python-2.7.3-setuptools-0.6c11'
New python executable in ./bin/python
Installing setuptools............done.
Installing pip...............done.

$ source ./bin/activate
(tahoe-install-test)
$ pip install allmydata-tahoe
Downloading/unpacking allmydata-tahoe
  Downloading allmydata-tahoe-1.10.0.tar.gz (1.4MB): 1.4MB downloaded
  Running setup.py egg_info for package allmydata-tahoe
    Not found: tahoe-deps
    Not found: ../tahoe-deps

    Installed /home/n/tmp/tahoe-install-test/build/allmydata-tahoe/Twisted-13.0.0-py2.7-linux-x86_64.egg
    Searching for zope.interface==3.6.0,==3.6.1,==3.6.2,>=3.6.5
    Reading http://pypi.python.org/simple/zope.interface/
    Best match: zope.interface 4.0.5
    Downloading https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-sdists/zope.interface-4.0.5.zip
    Processing zope.interface-4.0.5.zip
    Running zope.interface-4.0.5/setup.py -q bdist_egg --dist-dir /home/n/tmp/easy_install-B8abnY/zope.interface-4.0.5/egg-dist-tmp-7VsGgq
    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution

    Installed /home/n/tmp/tahoe-install-test/build/allmydata-tahoe/zope.interface-4.0.5-py2.7-linux-x86_64.egg

    package init file 'src/allmydata/web/static/__init__.py' not found (or not a regular file)
    package init file 'src/allmydata/web/static/css/__init__.py' not found (or not a regular file)
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Requirement already satisfied (use --upgrade to upgrade): setuptools>=0.6c6 in ./lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from allmydata-tahoe)
Downloading/unpacking zfec>=1.1.0 (from allmydata-tahoe)
  Downloading zfec-1.4.24.tar.gz (175kB): 175kB downloaded
  Running setup.py egg_info for package zfec

Downloading/unpacking simplejson>=1.4 (from allmydata-tahoe)
  Downloading simplejson-3.3.0.tar.gz (67kB): 67kB downloaded
  Running setup.py egg_info for package simplejson

Downloading/unpacking zope.interface==3.6.0,==3.6.1,==3.6.2,>=3.6.5 (from allmydata-tahoe)
  Downloading zope.interface-4.0.5.zip (173kB): 173kB downloaded
  Running setup.py egg_info for package zope.interface

    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
Downloading/unpacking Twisted>=11.0.0 (from allmydata-tahoe)
  Downloading Twisted-13.1.0.tar.bz2 (2.7MB): 2.7MB downloaded
  Running setup.py egg_info for package Twisted

Downloading/unpacking foolscap>=0.6.3 (from allmydata-tahoe)
  Downloading foolscap-0.6.4.tar.gz (479kB): 479kB downloaded
  Running setup.py egg_info for package foolscap

Downloading/unpacking pyOpenSSL (from allmydata-tahoe)
  Downloading pyOpenSSL-0.13.tar.gz (250kB): 250kB downloaded
  Running setup.py egg_info for package pyOpenSSL

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Downloading/unpacking Nevow>=0.6.0 (from allmydata-tahoe)
  Downloading Nevow-0.10.0.tar.gz (518kB): 518kB downloaded
  Running setup.py egg_info for package Nevow
    Traceback (most recent call last):
      File "<string>", line 16, in <module>
      File "/home/n/tmp/tahoe-install-test/build/Nevow/setup.py", line 3, in <module>
        from nevow import __version__ as version
      File "nevow/__init__.py", line 5, in <module>
        from nevow._version import version
      File "nevow/_version.py", line 2, in <module>
        from twisted.python import versions
    ImportError: No module named twisted.python
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/home/n/tmp/tahoe-install-test/build/Nevow/setup.py", line 3, in <module>

    from nevow import __version__ as version

  File "nevow/__init__.py", line 5, in <module>

    from nevow._version import version

  File "nevow/_version.py", line 2, in <module>

    from twisted.python import versions

ImportError: No module named twisted.python

----------------------------------------
Command python setup.py egg_info failed with error code 1 in /home/n/tmp/tahoe-install-test/build/Nevow
Storing complete log in /home/n/.pip/pip.log

[nejucomo]

This is a known Nevow vs pip issue: ​https://bugs.launchpad.net/nevow/+bug/1091055

[nejucomo]

I've submitted a patch to Nevow which fixes this issue (at least on my system): ​https://bugs.launchpad.net/nevow/+bug/1091055/comments/5

[nejucomo]

I consider this major because I'm part of the PyPI + pip hegemony which intends to take over the (python) world. Also, check out ​this claim of >40,000 PyPI downloads!

[nejucomo]

See also #1963 "replace nevow with twisted.web.template"

[nejucomo]

Replying to nejucomo:

I've submitted a patch to Nevow which fixes this issue (at least on my system): ​https://bugs.launchpad.net/nevow/+bug/1091055/comments/5

Also, to keep the ball rolling I attempted to translate the patch into a ​bzr / lp merge request.

[daira]

I believe that, due to ​this Nevow commit, the bug only happens when import setuptools fails (as it will in a clean virtualenv).

[daira]

... and Zooko correctly anticipated the problem ​two years ago. How annoying.

[daira]

[12:28] <dstufft> setuptools should always be importable when you're installing with pip (unless you hit the pip bug that uninstalls setuptools but your entire install will break then)

[12:31] <dstufft> daira1: pip is still broken with Nevow because the last release of Nevow was made in 2009 and it does not contain that commit

[12:32] <daira1> wow. okay

[nejucomo]

Replying to daira:

I believe that, due to ​this Nevow commit, the bug only happens when import setuptools fails (as it will in a clean virtualenv).

There may be more than one bug. I don't understand what the presence of setuptools has to do with the bug I find when running pip install nevow, which is that if setup.py imports nevow, then it will import twisted.

If twisted is not installed, then setup.py will raise an ImportError. This prevents a user from running "python ./setup.py <anything>" unless they've already installed a dependency. If this is true then I believe it would also break easy_install or any other case where setup.py is executed *prior* to satisfying dependencies.

[daira]

Replying to nejucomo:

Replying to daira:

I believe that, due to ​this Nevow commit, the bug only happens when import setuptools fails (as it will in a clean virtualenv).

There may be more than one bug. I don't understand what the presence of setuptools has to do with the bug I find when running pip install nevow, which is that if setup.py imports nevow, then it will import twisted.

I was mistaken: the Nevow commit isn't in any release yet [and isn't on trunk, since the branch was never merged]. pip depends on setuptools so that will always be present.

[zooko]

I've added a comment on ​https://bugs.launchpad.net/nevow/+bug/812537 saying that this issue is causing Tahoe-LAFS users to be unable to install it.

[warner]

Nevow-0.11.1 was released a few days ago (​https://github.com/twisted/nevow/blob/nevow-0.11.1/NEWS.txt), and supports install from 'pip'. I just confirmed that a simple pip install allmydata-tahoe in a clean virtualenv gets me a working tahoe binary (and tahoe debug repl can import nevow without complaint).

Closing this one out. Yay! Crediting this to the 1.11 milestone (even though it was fixed by something outside of tahoe's source tree) so we remember to mention it in the release notes.

[nejucomo]

Daira and I noticed we do not yet depend on Nevow >= 0.11.1. We're not yet sure if it's still possible to run into this problem, but it might be prudent to bump the requirement up.

I'm going to reopen this to ensure we bump the dependency version or discuss why not to do so.

[daira]

Replying to nejucomo:

Daira and I noticed we do not yet depend on Nevow >= 0.11.1. We're not yet sure if it's still possible to run into this problem, but it might be prudent to bump the requirement up.

I'm going to reopen this to ensure we bump the dependency version or discuss why not to do so.

Moved to #2291. The original problem is fixed just by the availability of Nevow 0.11.1.