updating setuptools with setup_requires causes build failure

[warner]

We had one set of build failures: the LGTM static-analysis tool was using the lowest-acceptable version of all dependencies, including setuptools-11.3, which was too old to understand the python_requires="<3.0" clause that tells the world how ashamed we are to not support py3. We fixed this by adding both an install_requires= and setup_requires= of >=28.8.0.

That caused a second set of build failures: when tox builds the sdist (before installing it into the new virtualenv), it does so with the sam e python that Tox itself uses (and using whatever version of setuptools was available to that python). Then it sees the setup_requires= in tahoe's setup.py and attempts to install it. Something doesn't get reloaded properly, because we wind up with an error that looks like:

tox -- --reporter=timing allmydata
 in dir /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build (timeout 1200 secs)
 watching logfiles {'test.log': '_trial_temp/test.log'}
 argv: ['tox', '--', '--reporter=timing', 'allmydata']
 environment:
  HOME=/var/lib/buildslave
  LOGNAME=buildbot
  PATH=/var/lib/buildslave/.local/bin:/usr/local/bin:/sbin:/usr/sbin:/bin:/usr/bin
  PWD=/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build
  SHELL=/bin/bash
  SHLVL=1
  TMPDIR=/var/folders/ql/2vmcxkls1tl1w71h9_1wvkch000035/T/
  USER=buildbot
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  XPC_FLAGS=0x0
  XPC_SERVICE_NAME=0
  _=/var/lib/buildslave/.local/bin/buildslave
  __CF_USER_TEXT_ENCODING=0x65:0:0
 using PTY: False
GLOB sdist-make: /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py
ERROR: invocation failed (exit code 1), logfile: /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/log/tox-0.log
ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python', local('/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/dist')]
env: None

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
  warnings.warn(msg)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)

Installed /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.eggs/setuptools-39.0.1-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 291, in <module>
    **setup_args
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py", line 272, in __init__
    _Distribution.__init__(self,attrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py", line 327, in finalize_options
    ep.load()(self, ep.name, value)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2370, in load
    return self.resolve()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2380, in resolve
    raise ImportError(str(exc))
ImportError: 'module' object has no attribute 'check_specifier'

ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py sdist --formats=zip --dist-dir /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/dist (see /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/log/tox-0.log)', 1)

This seems related to ​https://github.com/tox-dev/tox/issues/334

We think we can fix this with a workaround described in that ticket: add skipsdist = True to the [tox] section of tox.ini. Our [testenv] section already has skip_install = True because we (ab)use the deps feature to get editable installs (which are faster than building an sdist) with the "test" feature enabled (which gets us coverage and mock and stuff).

[testenv]
skip_install = True
deps = --editable=.[test]

Since we're already not using the default install step, I don't think we need that sdist tarball either. I'm hoping that the pip install used for the deps will use the new virtualenv's pip, which means it can upgrade setuptools (according to our setup.py's setup_requires=) without triggering this bug.

[Brian Warner <warner@…>]

In 526b97c/trunk:

tox: add 'skipsdist=True', hoping this will fix buildbot

There appears to be a bug in setuptools, triggered by running "python
setup.py sdist" with setuptools==11.3 in that python's environment, on a
project whose setup.py has a setup_requires= that requests setuptools >=
28.8.0. When setuptools is upgraded from inside setup.py, it gets into a
weird hybrid state where it's using setup() keyword-argument plugins from the
newer setuptools, but those plugins reference functions that aren't present
in the older setuptools, and the sdist command fails with an import
error (module object has no attribute 'check_specifier').

We don't actually need the sdist: all our tox test environments use
"skip_install = true", because we install tahoe via the "deps" line (so we
can get the [test] extra, and get a faster symlink-ish "editable" install).
That install uses "pip", which uses the pip inside the new virtualenv, which
either uses a newer version of setuptools (dependent upon what version of
"virtualenv" was installed in the parent environment, next to tox) or somehow
allows setuptools to be upgraded without exposing this weird broken hybrid
state.

Either way, skipping the sdist seems to fix this problem.

refs ticket:2910
​https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2910

[warner]

Yup, that seems to have fixed the problem.

I also had to change our Makefile to delegate tarball generation to tox, and add a tox environment to run python setup.py sdist as the Makefile used to do. This gives us a newer/updatable python/setuptools with which to run the sdist command. Without that change, the make tarballs target was failing with the same check_specifier exception from above.