[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2760: add dependency on `Twisted[tls]` to overcome pip's non-resolver

[Tahoe-LAFS]

#2760: add dependency on `Twisted[tls]` to overcome pip's non-resolver
-----------------------+------------------------
 Reporter:  warner     |          Owner:  warner
     Type:  defect     |         Status:  new
 Priority:  normal     |      Milestone:  1.11.0
Component:  packaging  |        Version:  1.10.2
 Keywords:             |  Launchpad Bug:
-----------------------+------------------------
 Tahoe itself doesn't strictly depend on TLS support. It depends on
 Foolscap, which *does*, but in a perfect world Tahoe wouldn't have to know
 about that.

 To ask Twisted to be TLS-capable, a package uses a square-bracketed
 "extra" in its dependencies, like {{{Twisted[tls] >= 15.2.1}}} instead of
 just {{{Twisted >= 15.2.1}}}.

 In that perfect world, we'd have:

 {{{
 Tahoe: Twisted >= 13.0.0, Foolscap
 Foolscap: Twisted[tls] >= 16.0.0
 }}}

 But unless/until we bypass the issue by using a requirements.txt file
 (maybe for #2055), we're affected by a missing pip feature: it lacks a
 full resolver.

 A full resolver is what lets Debian's "apt" try all possible combinations
 of packages and versions to find any (hopefully the "best") that will meet
 the given constraints. It takes a lot more work, and is scarily powerful
 (someone once proved that the constraint solver is Turing-complete).

 Pip installs the highest allowable version of the first thing that it
 encounters, and then explores the dependencies. It doesn't go back to try
 something different if the subsequent dependencies don't fit. And in
 particular, if it installs a package without any "extras", it won't go
 back and re-install it (with the extras) if it sees a later dependency
 that wants them.

 So in the above example, when we install tahoe, pip will first install the
 latest version of Twisted it can find (e.g. 16.0.0, with no extras), then
 it installs foolscap, then, it sees that Foolscap wants `Twisted[tls] >=
 16.0.0`. It knows it can satisfy the `>= 16.0.0` requirement, but it can't
 go back and re-install the `[tls]` extra. It emits a warning, but can't
 fix it.

 To overcome this, for the 1.11.0 release, we made Tahoe aware of the need
 for TLS, by using:

 {{{
 Tahoe: Twisted[tls] >= 13.0.0, Foolscap
 }}}

 But this hits another problem, which is that Twisted didn't start offering
 the `[tls]` extra until 15.2.1 . So we're actually using:

 {{{
 Tahoe: Twisted[tls] >= 15.2.1, Foolscap
 }}}

 even though Tahoe, itself, doesn't need that recent of a Twisted version.

 (incidentally, [https://foolscap.lothar.com/trac/ticket/249 foolscap#249]
 is what added `Twisted[tls]` to Foolscap)

 This version bump might be inconvenient for OS packagers who are
 backporting the new tahoe-1.11.0 to older distributions that don't have
 the more recent Twisted. As I mentioned on the [https://tahoe-
 lafs.org/pipermail/tahoe-dev/2016-March/009710.html mailing list] just
 now, porters who find themselves in this situation (and can't upgrade
 their Twisted packages) should consider modifying Tahoe's
 `src/allmydata/_auto_deps.py` to reduce this constraint back to the
 previous version. As long as there are OS-package-level dependency
 constraints on everything that the older version of Twisted needs for TLS
 support, things should work.

 (although note that the most recent version of Foolscap does, in fact,
 depend upon `Twisted >= 16.0.0`, so if you're upgrading that, you should
 probably go all-in and upgrade everything).

 This ticket is just to record the reasons for this version bump.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2760>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage