[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2077: pip packaging plan

[Tahoe-LAFS]

#2077: pip packaging plan
---------------------------+--------------------------------------
     Reporter:  daira      |      Owner:
         Type:  defect     |     Status:  new
     Priority:  normal     |  Milestone:  undecided
    Component:  packaging  |    Version:  1.10.0
   Resolution:             |   Keywords:  pip packaging setuptools
Launchpad Bug:             |
---------------------------+--------------------------------------

Comment (by warner):

 I'm in complete agreement with glyph and dstufft here (see #1951 and my
 experiments related to #2255). I think we can usefully distinguish between
 folks who simply want to *use* tahoe and those who want to take it to the
 next level and hack on it. The hackers can be expected to either know the
 basics of python hacking, or can be expected to read a second file to
 remind them of what their options are.

 Tahoe's packaging code dates back to 2007, before pip and virtualenv, at a
 time when setuptools was at best an interesting set of tradeoffs. We began
 with `setup.py build` and a Makefile that rummaged around in `build/lib*`
 to find the right thing to add to PYTHONPATH. I was eager to make the
 install process more GNU-like (`make install` into /usr/local, or `make
 install PREFIX=X` to use Stow or build a debian package), and there wasn't
 really a python-like process to use instead.

 These days, pip is well-established, setuptools is healthy and
 commonplace, and virtualenv is a great tool for providing the sort of
 isolated developer environment that we need.

 I don't feel that `download; unpack; setup.py build` needs to be the way
 people interact with tahoe. I'm not even sure what `setup.py build` is
 supposed to accomplish anymore: back in 2007 I thought it would create a
 `build/libSOMETHING` directory that could be put on PYTHONPATH (but even
 then it felt weird to parse Python's version just to figure out what the
 SOMETHING was).

 After 8 years, I think we're allowed to change the recommended command :).

 As glyph and dstufft have pointed out, our quickstart.rst is in fact aimed
 at developers: a non-developer won't bother to figure out wget/git-
 clone/tar to see the file in the first place. The furthest a non-developer
 can be expected to look is the front page of our website, which should
 have a pointer to the platform-specific installer or executable
 (#1951/#182/#195). We can put the user-oriented instructions on the
 website, and the developer-oriented instructions inside the source tree,
 and not be obligated to use the same text for both.


 There's one other feature I'd like to offer our hack-on-tahoe audience:
 safe builds (#2055). In particular, I'd like to offer a tool which fetches
 known versions of the dependencies (identified by SHA256) and makes it
 easy to build an environment that uses only those bits. The idea is to
 reduce the reliance set down to the original Tahoe source tree. (pip-over-
 https is a great start, but this would remove the pypi maintainers and the
 dependency authors from the set of people who could change the behavior of
 what you've built). It would require compilers and such, or hashes of
 wheels, but that's ok for the hacker audience.

 This could be a standalone script in the source tree, or an extension
 command in `setup.py`, but it should interact sensibly with the normal
 virtualenv tools. Like, the end goal for a developer is to have a local
 virtualenv with the right dependencies installed and a 'setup.py develop'
 symlink-ish in place. You might create this with "ve/bin/pip install
 tahoe-lafs" (which gives you the latest of everything) or this proposed
 "misc/safe-install" (which gives you the hash-verified compatible-but-
 older bits), but in either case you wind up with a ve/bin/tahoe that
 works.

 I have experimental branches in which we put a copy of pip and virtualenv
 in the source tree, enabling a standalone command to create this sort of
 environment. But if you're the sort of python hacker who already has them,
 you can just run your own. My thinking is that our quickstart.rst should
 point at this standalone tool, with a note that explains that it's really
 just running virtualenv+pip.

 Maybe we need that one-file executable to unblock this. I can imagine
 Zooko and Daira being more comfortable with making the source tree more
 developer-centric if they know there's a user-centric option that works.
 Doing it the other way around might feel riskier, or might feel like we're
 abandoning the non-developers.

 dstufft: could you add your !PyInstaller pointers/notes to #1951?

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2077#comment:26>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage