id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
2760	add dependency on `Twisted[tls]` to overcome pip's non-resolver	warner	warner	"Tahoe itself doesn't strictly depend on TLS support. It depends on Foolscap, which *does*, but in a perfect world Tahoe wouldn't have to know about that.

To ask Twisted to be TLS-capable, a package uses a square-bracketed ""extra"" in its dependencies, like {{{Twisted[tls] >= 15.2.1}}} instead of just {{{Twisted >= 15.2.1}}}.

In that perfect world, we'd have:

{{{
Tahoe: Twisted >= 13.0.0, Foolscap
Foolscap: Twisted[tls] >= 16.0.0
}}}

But unless/until we bypass the issue by using a requirements.txt file (maybe for #2055), we're affected by a missing pip feature: it lacks a full resolver.

A full resolver is what lets Debian's ""apt"" try all possible combinations of packages and versions to find any (hopefully the ""best"") that will meet the given constraints. It takes a lot more work, and is scarily powerful (someone once proved that the constraint solver is Turing-complete).

Pip installs the highest allowable version of the first thing that it encounters, and then explores the dependencies. It doesn't go back to try something different if the subsequent dependencies don't fit. And in particular, if it installs a package without any ""extras"", it won't go back and re-install it (with the extras) if it sees a later dependency that wants them.

So in the above example, when we install tahoe, pip will first install the latest version of Twisted it can find (e.g. 16.0.0, with no extras), then it installs foolscap, then, it sees that Foolscap wants `Twisted[tls] >= 16.0.0`. It knows it can satisfy the `>= 16.0.0` requirement, but it can't go back and re-install the `[tls]` extra. It emits a warning, but can't fix it.

To overcome this, for the 1.11.0 release, we made Tahoe aware of the need for TLS, by using:

{{{
Tahoe: Twisted[tls] >= 13.0.0, Foolscap
}}}

But this hits another problem, which is that Twisted didn't start offering the `[tls]` extra until 15.2.1 . So we're actually using:

{{{
Tahoe: Twisted[tls] >= 15.2.1, Foolscap
}}}

even though Tahoe, itself, doesn't need that recent of a Twisted version.

(incidentally, [https://foolscap.lothar.com/trac/ticket/249 foolscap#249] is what added `Twisted[tls]` to Foolscap)

This version bump might be inconvenient for OS packagers who are backporting the new tahoe-1.11.0 to older distributions that don't have the more recent Twisted. As I mentioned on the [https://tahoe-lafs.org/pipermail/tahoe-dev/2016-March/009710.html mailing list] just now, porters who find themselves in this situation (and can't upgrade their Twisted packages) should consider modifying Tahoe's `src/allmydata/_auto_deps.py` to reduce this constraint back to the previous version. As long as there are OS-package-level dependency constraints on everything that the older version of Twisted needs for TLS support, things should work.

(although note that the most recent version of Foolscap does, in fact, depend upon `Twisted >= 16.0.0`, so if you're upgrading that, you should probably go all-in and upgrade everything).

This ticket is just to record the reasons for this version bump.
"	defect	closed	normal	1.11.0	packaging	1.10.2	fixed