[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2913: PyPI "TLS<1.2 Brownout" is causing travis OS-X tests to fail

[Tahoe-LAFS]

#2913: PyPI "TLS<1.2 Brownout" is causing travis OS-X tests to fail
--------------------------------+------------------------
 Reporter:  warner              |          Owner:
     Type:  defect              |         Status:  new
 Priority:  normal              |      Milestone:  1.13.0
Component:  dev-infrastructure  |        Version:  1.12.1
 Keywords:                      |  Launchpad Bug:
--------------------------------+------------------------
 We're seeing some intermittent failures of the Travis-CI OS-X build, where
 the symptom is that tox is unable to install the "incremental" package
 (which is a dependency of Twisted):

 https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/359145018

 {{{
 Collecting Twisted[tls]>=16.4.0 (from tahoe-lafs==0.0.0)
   Downloading Twisted-17.9.0.tar.bz2 (3.0MB)
     Complete output from command python setup.py egg_info:
     Couldn't find index page for 'incremental' (maybe misspelled?)
     No local packages or working download links found for
 incremental>=16.10.1
         raise DistutilsError(msg)
     distutils.errors.DistutilsError: Could not find suitable distribution
 for Requirement.parse('incremental>=16.10.1')
 }}}

 http://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-
 tls-v12.html reports that PyPI is soon to be dropping support for TLS-1.0
 and 1.1, requiring all clients to use TLS-1.2 or newer. To test this,
 they're conducting rolling brownouts: as of today,
 https://status.python.org/ reports that TLS-1.2 is being enforced for the
 first 15 minutes of each hour (clients speaking older versions get an HTTP
 403 with an explanatory error message). Unfortunately many versions of
 pip/setuptools don't report these error messages very well, making them
 look like random network outages.

 OS-X-10.12 (the not-most-recent-version, which is Travis's default) ships
 with an ancient OpenSSL which does not support TLS-1.2 . OS-X-10.13 (the
 current version) ships with a modern-ish version of libressl that *can* do
 TLS-1.2.

 The system `/usr/bin/python` is linked against the system OpenSSL. Our
 Travis OS-X build appears to use that (although note that we don't have
 `language: python` turned on, for various reasons that need to be fixed,
 so we might get a different python if we told travis we wanted python).

 pip-9.0.3 knows how to use !OpenTransport on a mac, instead of OpenSSL,
 and OT is nicely modern and speaks TLS-1.2 just fine. But setuptools does
 not.

 So when TLS<=1.2 is turned off, the only way to install things on OS-X
 are:

 * be using OS-X-10.13 (the system libressl works), OR
 * be using a Homebrew python (which doesn't link against system openssl),
 OR
 * be using pip-9.0.3 or higher (which uses OpenTransport), OR

 In 4eac3ca, we modified our `setup_requires=` to need `setuptools >=
 28.8.0`, since that's (roughly?) the oldest that understands the
 `python_requires=` syntax that we use in our `setup.py`. However upgrading
 setuptools from inside a `setup_requires=` is pretty explody (https
 ://tahoe-lafs.org/buildbot-tahoe-
 lafs/builders/OS-X%2010.13/builds/14/steps/tox/logs/stdio), so in 6f20dbc
 we changed travis to upgrade setuptools before running tox, and in 526b97c
 we changed tox to stop building sdists (which we didn't use, and which
 were built with the external python and it's old setuptools).

 But the remaining problem is that when Twisted says `setup_requires:
 ["incremental"]`, it's setuptools that attempts to do the install, not
 pip. Since Travis is using OS-X 10.12, and the system python, and it's not
 using pip, the setuptools attempt to install `incremental` uses an old
 version of TLS, which gets blocked by the PyPI brownout, and unhelpfully
 reported as a lookup failure.

 To fix this, our workaround will be to pre-install/upgrade `incremental`,
 in our tox.ini. We're going to need to pre-install anything that is
 referenced by `setup_requires` in any dependency. We can probably remove
 this workaround when Travis moves their default to OS-X-10.13, or if
 setuptools acquires the same kind of workaround that pip has (not likely),
 or if we switch to telling travis `language: python` and that happens to
 use something like Homebrew python.

 Travis is having other problems right now (OS-X builds are backed up
 pretty badly), but I *think* this TLS thing is what's biting us. It
 certainly makes it harder to experiment, though.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2913>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage