[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2378: the test-from-egg buildstep has started failing on all builders

[Tahoe-LAFS]

#2378: the test-from-egg buildstep has started failing on all builders
-------------------------+-------------------------------------------------
     Reporter:  daira    |      Owner:  daira
         Type:  defect   |     Status:  assigned
     Priority:  major    |  Milestone:  1.10.1
    Component:           |    Version:  1.10.0
  packaging              |   Keywords:  buildbot test-from-egg blocks-
   Resolution:           |  release
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Comment (by warner):

 I was able to reproduce this locally, eventually, with the following
 (paraphrased slightly):

 1. cd ~/tahoe (contains setuptools.egg, really zetuptoolz)
 2. python setup.py bdist_egg  -> produces "tahoe.egg"
 3. mkdir ~/tmp ~/tmp/egginst
 4. cd ~/tmp
 5. PYTHONPATH=egginst easy_install -d egginst ~/tahoe/dist/tahoe.egg
 6. cd egginst
 7. PYTHONPATH=~/tahoe/setuptools.egg:./*.egg python tahoe debug trial

 The PYTHONPATH in step 5 is necessary to appease `easy_install`, which is
 trying to protect you against writing into a directory that you wouldn't
 normally read from. The PYTHONPATH in step 7 does two things: it uses
 tahoe's zetuptoolz fork, and it makes the dependencies available.

 Step 7 produces the "Missing distribution spec" exception.

 The `egginst` directory, as populated by `easy_install`, contains an .egg
 for every dependency, an `easy-install.pth` which would add them to the
 path (if you included `egginst` on PYTHONPATH), and an executable entry-
 point script for everything that produced one (tahoe, flappserver,
 flogtool, some zfec ones, etc). It is this `tahoe` entrypoint script that
 we execute. On my machine, it contains the following:

 {{{
 #!/usr/local/opt/python/bin/python2.7
 # EASY-INSTALL-ENTRY-SCRIPT: 'allmydata-
 tahoe==1.10.0.post272','console_scripts','tahoe'
 __requires__ = 'allmydata-tahoe==1.10.0.post272'
 import sys
 from pkg_resources import load_entry_point

 if __name__ == '__main__':
     sys.exit(
         load_entry_point('allmydata-tahoe==1.10.0.post272',
 'console_scripts', 'tahoe')()
     )
 }}}

 (this matches what I'm seeing on freestorm's buildslave).

 If I omit tahoe's zetuptoolz egg from the PYTHONPATH in step 7, the
 problem goes away, and the tests run normally.

 Note that step 5 is using my system-supplied `easy_install`, which reports
 its `--version` as `setuptools 12.1`. I think the problem here is a
 mismatch between the version of setuptools (zetuptoolz) that produced the
 .egg file in step 2, the version of easy_install that installed the egg
 (setuptools-12.1) in step 5, and the version of setuptools (zetuptoolz
 again) which provides the `pkg_resources` used for execution in step 7.

 Our toolchain takes great pains to work even if you don't have setuptools
 installed on your system (perhaps due to my early complaints about not
 liking setuptools :-), by providing a setuptools/zetuptoolz .egg in the
 source tree, and importing it early in `setup.py`. However, step 5 in the
 test-from-egg process depends upon having a system-supplied `easy_install`
 script. (Tahoe includes setuptools as an egg, but not easy_install).

 Digging into the zetuptoolz `pkg_resources`, it seems that the single-
 string `__requires__` in the entrypoint script is, ''sometimes'', being
 interpreted as a list, causing it to parse each letter of the string as a
 separate requirement string. Most of these ("a", "l", "l", "m", etc) are
 ignored, but when it finally gets to "=", the parser throws an exception,
 because "=" is supposed to have a package name on the left. However, in
 other places within `pkg_resources`, the string is correctly interpreted
 as a string.

 I've seen other entrypoint scripts that provide a list for `__requires__`,
 like the one tahoe's `setup.py build` installs into `support/bin/tahoe`:

 {{{
 #!/usr/local/opt/python/bin/python2.7
 # EASY-INSTALL-ENTRY-SCRIPT: 'allmydata-
 tahoe==1.10.0.post272.dev0','console_scripts','tahoe'
 # generated by zetuptoolz 0.6c16dev5
 __requires__ = ['allmydata-tahoe==1.10.0.post272.dev0',
 'setuptools>=0.6c6', 'zfec>=1.1.0', 'simplejson>=1.4',
 'zope.interface>=3.6.0,!=3.6.3,!=3.6.4', 'foolscap>=0.6.3',
 'pycrypto>=2.1.0,!=2.2,!=2.4', 'mock>=0.8.0', 'pycryptopp>=0.6.0',
 'Twisted>=13.0.0', 'Nevow>=0.11.1', 'service-identity',
 'characteristic>=14.0.0', 'pyasn1>=0.1.4', 'pyasn1-modules',
 'pyOpenSSL>=0.13', 'cryptography', 'cffi>=0.8', 'six>=1.4.1', 'enum34',
 'pycparser']
 import sys
 from pkg_resources import load_entry_point

 sys.exit(
    load_entry_point('allmydata-tahoe==1.10.0.post272.dev0',
 'console_scripts', 'tahoe')()
 )
 }}}

 And if I make `easy_install` use tahoe's zetuptoolz egg, by replacing step
 5 with:

 5b. PYTHONPATH=~/tahoe/setuptools.egg:egginst easy_install -d egginst
 ~/tahoe/dist/tahoe.egg

 then I get the list-`__requires__` entrypoint script like the one in
 `support/bin/tahoe`.

 So my hunches are:

 * zetuptoolz produces, and requires, a `__requires__` list
 * in modern setuptools:
   * `pkg_resources` accepts a string or a list
   * `easy_install` produces a string

 and the mismatch is between the version of easy_install that creates the
 entrypoint script, and the version of pkg_resources that gets loaded by
 that script.

 == Solutions ==

 So one solution for this ticket might be to use zetuptoolz for the
 `easy_install`, like step 5b. I'm experimenting with this to see if it
 would work.

 A second would be to fix zetuptoolz to look more like modern setuptools.

 But I suspect this test is no longer exercising the kind of functionality
 we really care about. Nobody installing a tahoe .egg is going to use 5b:
 they'll use `easy_install tahoe.egg`. And they won't include tahoe's
 zetuptools in PYTHONPATH when they run tahoe, so they'll get their
 pkg_resources from the system setuptools. To be relevant, our test should
 match real-world usage.

 So a third fix is to remove the non-realistic PYTHONPATH= from step 7. The
 `easy-install.pth` in egginstalldir means we could probably use:

 7b. PYTHONPATH=~/tmp/egginst python tahoe debug trial

 (This reflects reality, since eggs are usually installed into something on
 the built-in PYTHONPATH, like /usr/local/lib).

 Doing that would make the test dependent upon a system-installed
 setuptools, but it was already depending on that for easy_install.

 I'm going to try this third option and see if it works.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2378#comment:6>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage