#2913 closed defect (fixed)

PyPI "TLS<1.2 Brownout" is causing travis OS-X tests to fail

Reported by: warner Owned by: Brian Warner <warner@…>
Priority: normal Milestone: 1.13.0
Component: dev-infrastructure Version: 1.12.1
Keywords: Cc:
Launchpad Bug:

Description

We're seeing some intermittent failures of the Travis-CI OS-X build, where the symptom is that tox is unable to install the "incremental" package (which is a dependency of Twisted):

https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/359145018

Collecting Twisted[tls]>=16.4.0 (from tahoe-lafs==0.0.0)
  Downloading Twisted-17.9.0.tar.bz2 (3.0MB)
    Complete output from command python setup.py egg_info:
    Couldn't find index page for 'incremental' (maybe misspelled?)
    No local packages or working download links found for incremental>=16.10.1
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('incremental>=16.10.1')

http://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.html reports that PyPI is soon to be dropping support for TLS-1.0 and 1.1, requiring all clients to use TLS-1.2 or newer. To test this, they're conducting rolling brownouts: as of today, https://status.python.org/ reports that TLS-1.2 is being enforced for the first 15 minutes of each hour (clients speaking older versions get an HTTP 403 with an explanatory error message). Unfortunately many versions of pip/setuptools don't report these error messages very well, making them look like random network outages.

OS-X-10.12 (the not-most-recent-version, which is Travis's default) ships with an ancient OpenSSL which does not support TLS-1.2 . OS-X-10.13 (the current version) ships with a modern-ish version of libressl that *can* do TLS-1.2.

The system /usr/bin/python is linked against the system OpenSSL. Our Travis OS-X build appears to use that (although note that we don't have language: python turned on, for various reasons that need to be fixed, so we might get a different python if we told travis we wanted python).

pip-9.0.3 knows how to use OpenTransport on a mac, instead of OpenSSL, and OT is nicely modern and speaks TLS-1.2 just fine. But setuptools does not.

So when TLS<=1.2 is turned off, the only way to install things on OS-X are:

  • be using OS-X-10.13 (the system libressl works), OR
  • be using a Homebrew python (which doesn't link against system openssl), OR
  • be using pip-9.0.3 or higher (which uses OpenTransport?), OR

In 4eac3ca, we modified our setup_requires= to need setuptools >= 28.8.0, since that's (roughly?) the oldest that understands the python_requires= syntax that we use in our setup.py. However upgrading setuptools from inside a setup_requires= is pretty explody (https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/OS-X%2010.13/builds/14/steps/tox/logs/stdio), so in 6f20dbc we changed travis to upgrade setuptools before running tox, and in 526b97c we changed tox to stop building sdists (which we didn't use, and which were built with the external python and it's old setuptools).

But the remaining problem is that when Twisted says setup_requires: ["incremental"], it's setuptools that attempts to do the install, not pip. Since Travis is using OS-X 10.12, and the system python, and it's not using pip, the setuptools attempt to install incremental uses an old version of TLS, which gets blocked by the PyPI brownout, and unhelpfully reported as a lookup failure.

To fix this, our workaround will be to pre-install/upgrade incremental, in our tox.ini. We're going to need to pre-install anything that is referenced by setup_requires in any dependency. We can probably remove this workaround when Travis moves their default to OS-X-10.13, or if setuptools acquires the same kind of workaround that pip has (not likely), or if we switch to telling travis language: python and that happens to use something like Homebrew python.

Travis is having other problems right now (OS-X builds are backed up pretty badly), but I *think* this TLS thing is what's biting us. It certainly makes it harder to experiment, though.

Change History (2)

comment:1 Changed at 2018-03-29T01:40:50Z by Brian Warner <warner@…>

In acc2b57/trunk:

tox: use newer (tox-2.4) settings, pre-install 'incremental'

  • use 'extras' for our [test] additions instead of abusing 'deps'
  • use 'deps' to pre-install 'incremental', which we couldn't do when we filled it with --editable to get [test]
  • pre-install 'incremental' to work around a bug that strikes on Travis under OS-X-10.12 as PyPI gradually disables TLS<1.2. See ticket 2913 for details
  • remove redundant configuration lines
  • require tox-2.4 or newer, to get 'extras'

refs ticket:2913
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2913

comment:2 Changed at 2018-03-29T01:40:51Z by Brian Warner <warner@…>

  • Owner set to Brian Warner <warner@…>
  • Resolution set to fixed
  • Status changed from new to closed

In 479588d/trunk:

Merge branch '2913-travis-osx'

closes ticket:2913

Note: See TracTickets for help on using tickets.