[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2193: pyOpenSSL 0.14 pulls in a bunch of new dependencies

[Tahoe-LAFS]

#2193: pyOpenSSL 0.14 pulls in a bunch of new dependencies
-------------------------+-------------------------------------------------
     Reporter:  daira    |      Owner:  unassigned
         Type:  defect   |     Status:  new
     Priority:  major    |  Milestone:  1.11.0
    Component:           |    Version:  1.10.0
  packaging              |   Keywords:  packaging setuptools pyopenssl
   Resolution:           |  cryptography six cffi pycparser
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Comment (by zooko):

 Replying to [comment:28 daira]:
 > Replying to [comment:27 gdt]:
 > > I'm not entirely following the pinning argument.  tahoe is building ok
 in pkgsrc with 0.14.  So if you require 0.13, the package will fail and no
 longer be available, and I'll just mark it BROKEN.   What's wrong with a
 situation where pyOpenSSL 0.14 is properly installed, with all its
 dependencies?
 >
 > That's a good question. The issue is that the Python package-
 distribution metadata conflates the question "what dependent package
 versions are acceptable to meet this package's requirement?", with the
 question "what dependent package versions should I attempt to fetch and
 build in order to meet this package's requirement (when not using an OS
 packaging system)?" That's a problem in situations where the fetch-and-
 build process is unreliable.
 >
 > We ''could'' hack around this conflation by doing the following:
 > * From Tahoe's `setup.py`, attempt "`from OpenSSL import SSL`".
 > * If that succeeds and passes version sanity checks (including checks on
 the linked OpenSSL version for #2215), then use the requirement `pyOpenSSL
 == $WHATEVER_VERSION_WAS_IMPORTED`.
 > * If that does not succeed or does not pass version sanity checks, then
 use the requirement `pyOpenSSL == 0.13` (or `== 0.13.1` if we decide to do
 the OpenSSL version check that way).
 >
 > That would implement the following:
 >
 > > If the new code handles that gracefully, there's no issue - if all
 you're talking about is forcing a choice of 0.13 when the build of
 pyOpenSSL is triggered from within tahoe-lafs that sounds fine.
 >
 > However, attempting to import dependent packages from `setup.py` has
 caused problems in the past and I'm not entirely sure it's a good idea. If
 we do it, we should have some exit strategy to stop doing it in a future
 Tahoe-LAFS version.


 I don't think we should add all this logic into the Tahoe-LAFS build
 system. I agree that there is a problem here, but I don't think we should
 try to address that problem by adding code into the Tahoe-LAFS build
 system. The problem, as I understand it, is that there are two different
 audiences listening to Tahoe-LAFS's announcements about what version of
 `pyOpenSSL` it depends on. One audience is packaging engineers like gdt,
 and the other is the automatic build mechanisms such as `pip`.

 There is no single statement we can make that will give the right idea to
 both of these audiences. So I propose that we let the machine-parseable
 declaration in
 [source:trunk/src/allmydata/_auto_deps.py?annotate=blame&rev=7bb07fb5e28756fa13ba5190e6c39003c84d3e1e
 src/allmydata/_auto_deps.py] be optimized for the automatic build
 mechanisms like `pip`, and we use an attendant comment to communicate to
 the human engineer like gdt. So I propose a solution along these lines:

 {{{
 diff --git a/src/allmydata/_auto_deps.py b/src/allmydata/_auto_deps.py
 index b7be321..93a9994 100644
 --- a/src/allmydata/_auto_deps.py
 +++ b/src/allmydata/_auto_deps.py
 @@ -38,6 +38,9 @@ install_requires = [
      # * foolscap < 0.6.3 is incompatible with Twisted-11.1.0 and newer.
 Since
      #   current Twisted is 12.0, any build which needs twisted will grab
 a
      #   version that requires foolscap>=0.6.3
 +    #
 +    "foolscap >= 0.6.3",
 +
      # * pyOpenSSL is required by foolscap for it (foolscap) to provide
 secure
      #   connections. Foolscap doesn't reliably declare this dependency in
 a
      #   machine-readable way, so we need to declare a dependency on
 pyOpenSSL
 @@ -47,8 +50,21 @@ install_requires = [
      #   pyOpenSSL, or foolscap switching from pyOpenSSL to a different
 crypto
      #   library, we need to update this declaration here.
      #
 -    "foolscap >= 0.6.3",
 -    "pyOpenSSL",
 +    # * pyOpenSSL >= 0.14 uses a new implementation which depends on the
 new
 +    #   "cryptography.io" library, which depends on cffi and other *
 +    #   things. Our automated build system (as run by typical users on
 their
 +    #   * typical systems, which may not have a compiler or Python header
 +    #   files * installed), can't handle those new dependencies, so for
 now
 +    #   we're pinning this dependency to pyOpenSSL == 0.13.
 +    #
 +    #   Note that if you are a build engineer who already has pyOpenSSL
 >=
 +    #   v0.14 working and wants to use Tahoe-LAFS with pyOpenSSL >=
 v0.14, go
 +    #   right ahead! You'll have to edit or comment-out this pinning
 +    #   declaration here. This declaration is not for your benefit, but
 for
 +    #   the benefit of the automatic build system (i.e. pip or
 easy_install)
 +    #   as used by our non-expert users.
 +    #
 +    "pyOpenSSL == 0.13",

      "Nevow >= 0.6.0",

 }}}

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2193#comment:35>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage