[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2735: remove "bin/tahoe" and fancy "@" runner support

[Tahoe-LAFS]

#2735: remove "bin/tahoe" and fancy "@" runner support
----------------------------+---------------------------
 Reporter:  warner          |          Owner:
     Type:  task            |         Status:  new
 Priority:  normal          |      Milestone:  undecided
Component:  code-nodeadmin  |        Version:  1.10.2
 Keywords:                  |  Launchpad Bug:
----------------------------+---------------------------
 == There And Back Again: Climbing The Complexity Mountain (and Coming Back
 Down The Other Side)

 A long long time ago, one of the first bits of node-management code we
 added to the Tahoe tree was a launcher script which set PYTHONPATH (to the
 source tree), then ran `twistd` to load a tahoe node from a .tac file
 (which daemonized into the background).

 Later ([b75276a] in Sep-2007), when we were figuring out dependency
 management, we ended up writing a bespoke implementation of virtualenv
 (before virtualenv was a thing). This was a scheme where everything we
 needed (except tahoe itself) was installed into a `support/` directory,
 and the launcher script's $PYTHONPATH pointed into both `support/` and the
 source tree's `src/` directory (for tahoe). Because the `twistd` binary
 had to come from `support/bin/`, the launcher script had to set $PATH too.

 This launcher script was installed as `bin/tahoe` at the top of the source
 tree, so the build process was like `python setup.py build` and then
 `bin/tahoe --version`. Getting the shbang line right hard, as was making
 sure the file had a windows ".exe" suffix when appropriate, so ([083795d]
 in Jan-2009) `bin/tahoe` was generated from a template, instead of being
 checked in like a regular file, which introduced a new `make_executable`
 command for setup.py.

 About that same time ([b77c89a] in Jan-2009), some unit tests appeared,
 which had complicated checks to make sure that $PATH contained the
 expected things, and that we weren't executing `tahoe` from anywhere else,
 etc. The unit tests were made more challenging by the fork-and-exec that
 occurred inside `twistd`, which also escapes control of the code-coverage
 tracing function. To allow the `exec()` to locate twistd, both $PATH and
 $PYTHONPATH must be inherited by all child processes, so by this point the
 command to run the tahoe test suite had blossomed from a simple `trial
 allmydata` to a complex shell script.

 The pinnacle of complexity was achieved ([3798d99] in Jan-2011) when the
 `tahoe` executable itself absorbed responsibility for running the test
 suite, or `coverage`, or `flogtool`, or `python`, or other commands that
 might need $PATH/$PYTHONPATH set to fully immerse oneself in the Tahoe
 lifestyle. `python bin/tahoe debug repl` was just like `python` but
 different. `python bin/tahoe @python -3 tahoe --version` let you put extra
 python arguments into your tahoe. `python bin/tahoe @coverage run @tahoe
 debug trial allmydata` became a thing.

 == It's All Downhill From Here

 Fortunately, by that time ([ac3b26ec] in Oct-2010), we had already stopped
 shelling out to a `twistd` executable: instead, we imported the twistd
 library and invoked its functionality directly. This retains the `fork()`,
 but removes the `exec()`. Still later ([87a6894] in May-2012) we stopped
 using the contents of .tac files to load the tahoe node.

 It's time to continue the trend towards simplicity. I believe we no longer
 need anything which manipulates $PATH or $PYTHONPATH, and can instead rely
 upon a fully-installed tahoe+dependencies (either installed to the system
 "for reals", or in a virtualenv). The one entry point shall be a bare
 "tahoe" installed as a setup.py "entrypoint", and looked up by the shell
 on $PATH like any other command. `trial allmydata`, perhaps fronted by
 `tox` (or run under `coverage`), shall be the one test-running command.
 `pip install .` (perhaps with `--editable`) shall be the one install-from-
 source-tree command.

 == Simplify! Simplify! Simplify!

 To that end, I'd like to delete the following:

 * `bin/tahoe` and `setup.py make_executable`
 * `scripts.debug.repl`, `trial`, `flogtool`, and all the "@" stuff
 * `test_runner.BinTahoe`
 * `test_runner.RunNode`

 (although maybe `test_runner.BinTahoe.test_unicode_arguments_and_output`,
 which makes sure that unicode values in argv get delivered to the tahoe
 code correctly, should be retained in some form that doesn't require so
 much complexity in the entrypoint script)

 We need to keep exercising the `tahoe start` functionality in unit tests.
 This is tricky because `twistd`, even as a library, terminates the calling
 process, so we need a `fork()` in the test code. But I think we can write
 a narrow test that doesn't require `tahoe` to be a general-purpose binary
 launcher.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2735>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage