[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2077: pip packaging plan

[Tahoe-LAFS]

#2077: pip packaging plan
---------------------------+--------------------------------------
     Reporter:  daira      |      Owner:
         Type:  defect     |     Status:  new
     Priority:  normal     |  Milestone:  undecided
    Component:  packaging  |    Version:  1.10.0
   Resolution:             |   Keywords:  pip packaging setuptools
Launchpad Bug:             |
---------------------------+--------------------------------------

Comment (by glyph):

 Sorry for the monster comment length here; please read at your leisure,
 and if there's important release stuff going on, do not feel the need to
 address this now.  I'm trying to collect all of my thoughts about this
 into one place to leave as few assumptions unstated as possible.

 I am trying to understand the desired user experience outlined on this
 ticket.

 Here's the experience I want: as a developer and system administrator, I
 want to install Tahoe to hack on it the same way I install anything else
 that uses Python: `pip wheel tahoe-lafs; pip install tahoe-lafs`.  To work
 on the code directly I want to be able to do `mkvirtualenv Tahoe; pip
 install -e .`.  This works for all the other Python stuff I work on and
 use.  It appears that if I do this right now (via `pip wheel allmydata-
 tahoe`) I get a version that is 3 years out of date which dies with a
 traceback: `distutils.errors.DistutilsError: Could not find suitable
 distribution for
 Requirement.parse('zope.interface==3.6.0,==3.6.1,==3.6.2,>=3.6.5')`.

 This ticket seems to propose an extraordinarily complicated way of
 achieving this result, by bundling specific versions of pip and setuptools
 (I guess to preserve their present vulnerabilities in perpetuity within
 Tahoe? :)) rather than having users retrieve `get-pip.py` et. al. from
 [https://pip.pypa.io/en/stable/ the usual place].  Reading between the
 lines, my understanding of this solution is that it intends to provide a
 quicker, one-step on-boarding process for developers who have never worked
 on another Python project, and want to get started with Tahoe without
 understanding how the packaging ecosystem works.  I don't think this works
 very well; much as the packaging ecosystem needs to improve, it provides a
 far better and far better-documented experience than the "helpers" Tahoe
 provides to get you started working on it.

 As a user, I don't want to touch pip or setuptools or for that matter a
 Python interpreter at all; I want to double-click on a mac app and be
 presented with a UI.  Tahoe does not have any build solution for this sort
 of user right now, so I think it can be ignored for the purposes of this
 ticket.

 Some users may want to install e.g. a Debian package, but that's Debian's
 responsibility to build, and they have an extensive toolchain that can be
 applied to anything that works with `pip install`.

 So here's my straw-man proposal:

   1. Delete `zetuptoolz` entirely.
   2. Delete all the code in `setup.py` to reduce interactions with `pip`
 and tools like it running `setup.py`; "helpers" like `setup.py trial` can
 be run directly by developers.  Similarly, eliminate the "sumo" option;
 it's not clear to me who this helps.
   3. Replace the `setup.py` with a small, declarative example, something
 like [https://github.com/hynek/doc2dash/blob/master/setup.py this] (ignore
 the py.test command that contradicts point 2 ;-)), which just says
 `install_requires`, and possibly syncs the version or uses something like
 Brian's [https://pypi.python.org/pypi/versioneer/ versioneer].  Use
 `entry_points` to produce console scripts rather than the bespoke
 `MakeExecutable`.  To get the "required security" mentioned in the
 description, don't bundle setuptools, simply add a version check to the
 top of `setup.py` and refuse to run with antediluvian versions.
   4. Replace half of `quickstart.rst` with a line that just says
 `virtualenv tahoe; tahoe/bin/pip install tahoe-lafs`.

 This does leave the issue of not trusting setuptools to put the right
 versions in the right places on `sys.path`.  If you tell users to create
 new, normal (i.e. the default, not `--system-site-packages`) virtual
 environments, rather than trying (and, based on the long history of
 painful ticket comments in this cluster of tickets, consistently failing)
 to convince Python to use a set of modules built into a particular
 directory as its primary import location, you skip not only all the
 confusing behavior in setuptools and distutils, but also quite a lot of
 confusing behavior in Python itself around which directories take
 precedence under which conditions.  Virtual environments provide a
 predictable execution context which is almost impossible to ''completely''
 replicate with a Python interpreter environment that has arbitrary other
 stuff installed in it.

 There's also the issue of the `console_scripts` entry point executable
 shims not doing the right thing on Windows.  I saw a previous comment
 about it not working on win64; [https://bitbucket.org/pypa/setuptools
 /history-node/11a96ddefa71/launcher.c?at=default this has apparently been
 fixed].  I'm not sure about the unicode command-line arguments issue but
 [https://bitbucket.org/pypa/setuptools/issues?q=unicode I don't see an
 issue on their tracker about it].  One of the reasons to prefer this
 solution on Windows, by the way, is that if you don't have python 3
 installed (and the `py.exe` launcher) then shebang lines won't be honored
 in `.py` (or `.pyscript`) files and they'll be launched with the global
 python interpreter.  There's only one global setting for what runs
 "Python.File" files, and on my Windows computers it's generally a text
 editor of some kind, so the tahoe command-line just doesn't work.  (For
 what it's worth, neither does Twisted's, and I am hoping we switch to
 entry points for this reason as well.)

 I did say that this was a straw-man proposal though, and that's because I
 am pretty sure I'm missing some requirements that Tahoe has.  I strongly
 suspect that these other requirements would be better-satisfied by scripts
 that sit as wrappers ''around'' `setup.py` rather than trying to burrow
 around ''inside'' of distutils and setuptools, but I'd have to know what
 they are to be sure of that.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2077#comment:17>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage