1.12.0b1 tagged: please test!

[Brian Warner]

I finished editing the NEWS file, and did a general cleanup pass on the
docs/ directory, so I think everything is now in shape for our first
beta. 1.12.0b1 was just tagged (and signed with the Tahoe-LAFS Release
Key, E34E 62D0 6D0E 69CF CA41 79FF BDE0 D31D 6866 6A7A), so the tarballs
are available in the usual place:

 https://tahoe-lafs.org/source/tahoe-lafs/tarballs/

Now is the time to discover the last-minute bugs and packaging problems.
Target for the final release is next weekend.

Thanks to everyone testing out this release!
 -Brian


PS: here's a preview of the NEWS file, describing what's new in this
release


New Features
------------

This release features improved Tor/I2P integration. It is now easy to::

* use Tor to hide your IP address during external network activity
* connect to Tor/I2P-based storage servers
* run an Introducer or a storage node as a Tor "onion service"

See docs/anonymity-configuration.rst for instructions and new
node-creation arguments (--hide-ip, --listen=tor), which include ways to
use SOCKS servers for outbound connections. Tor/I2P/Socks support
requires extra python libraries to be installed (e.g. 'pip install
tahoe-lafs[tor]'), as well as matching (non-python) daemons available on
the host system. (tickets #517,
#2490, #2838)

Nodes can use multiple introducers by adding entries to a new
``private/introducers.yaml`` file, or stop using introduction entirely
by omitting the ``introducer.furl`` key from tahoe.cfg (introducerless
clients will need static servers configured to connect anywhere). Server
announcements are sent to all connected Introducers, and clients merge
all announcements they see, which can improve grid reliability. (#68)

In addition, nodes now cache the announcements they receive in a YAML
file, and use their cached information at startup until the Introducer
connection is re-established. This makes nodes more tolerant of
Introducers that are temporarily offline. Nodes admins can copy text
from the cache into a new ``private/servers.yaml`` file to add "static
servers", which augment/override what the Introducer offers. This can
modify aspects of the server, or use servers that were never announced
in the first place. (#2788)

Nodes use a separate Foolscap "Tub" for each server connection, so
``servers.yaml`` can override the connection rules (Tor vs direct-TCP)
for each one separately. This offers a slight privacy improvement, but
slows down connections slightly (perhaps 75ms per server), and breaks an
obscure NAT-bypass trick which enabled storage servers to run behind NAT
boxes (but only when all the *clients* of the storage server had public
IP addresses, and were also configured as servers). (#2759, #517)

"Magic Folders" is an experimental two-way directory synchronization
tool, contributed by Least Authority Enterprises, which replaces the
previous experimental (one-way) "drop-upload" feature. This allows
multiple users to keep a single directory in-sync, using Tahoe as the
backing store. See docs/frontends/magic-folder.rst for details and
configuration instructions.

Compatibility Issues
--------------------

The old version-1 Introducer protocol has been removed. Tahoe has used
the version-2 protocol since 1.10 (released in 2013), but all nodes
(clients, servers, and the Introducer itself) provided
backwards-compatibility translations when encountering older peers.
These translations were finally removed, so Tahoe nodes at 1.12 or later
will not be able to interact with nodes at 1.9 or older. (#2784)

The versions of Tahoe (1.11.0) and Foolscap (0.6.5) that shipped in
Debian/Jesse (the most recent stable release, as of December 2016) are
regrettably not forwards-compatible with the new release. Nodes running
Jesse will not be able to connect to servers or introducers created with
this release: they cannot parse the new ``tcp:HOST:PORT`` hint syntax
(this syntax has been around for a while, but this is the first Tahoe
release to automatically generate such hints). If you need to work
around this, then after creating your new node, edit the tahoe.cfg of
your new server/introducer: in ``[node] tub.location``, make each
connection hint look like ``HOST:PORT`` instead of ``tcp:HOST:PORT``. If
your grid only has nodes with Foolscap-0.7.0 or later, you will not need
this workaround. (#2831)

Nodes now use an Ed25519 public key as a serverid, instead of a Foolscap
"tub id", so status displays will report a different serverid after
upgrade. For the most part this should be self-consistent, however if
you have an old (1.11) client talking to a new (1.12) Helper, then the
client's upload results (on the "Recent Uploads And Downloads" web page)
will show unusual server ids. (#1363)

Dependency/Installation changes
-------------------------------

Tahoe now requires Twisted >= 16.1.0, so ensure that unit tests do not
fail because of uncancelled timers left running by HostnameEndpoint. It
also requires the Tor/I2P supporting code from Foolscap >= 0.12.5
. (#2781)

Configuration Changes
---------------------

Some small changes were made to the way Tahoe-LAFS is configured, via
``tahoe.cfg`` and other files. In general, node behavior should now be
more predictable, and errors should be surfaced earlier.

* ``tub.port`` is now an Endpoint server specification string (which is
  pretty much just like a strports string, but can be extended by
  plugins). It now rejects "tcp:0" and "0". The tahoe.cfg value
  overrides anything stored on disk (in client.port). This should have
  no effect on most old nodes (which did not set tub.port in tahoe.cfg,
  and which wrote an allocated port number to client.port the first time
  they launched). Folks who want to listen on a specific port number
  typically set tub.port to "tcp:12345" or "12345", not "0". (ticket
  #2491)
* This should enable IPv6 on servers, either via AAAA records or
  colon-hex addresses. (#2827)
* The "portnumfile" (e.g. NODEDIR/client.port) is written as soon as the
  port is allocated, before the tub is created, and only if "tub.port"
  was empty. The old code wrote to it unconditionally, and after Tub
  startup. So if the user allows NODEDIR/client.port to be written, then
  later modifies tahoe.cfg to set "tub.port" to a different value, this
  difference will persist (and the node will honor tahoe.cfg "tub.port"
  exclusively).
* We now encourage static allocation of tub.port, and pre-configuration
  of the node's externally-reachable IP address or hostname (by setting
  tub.location). Automatic IP-address detection is deprecated. Automatic
  port allocation is discouraged. Both are managed by the new arguments
  to "tahoe create-node".
* "tahoe start" now creates the Tub, and all primary software
  components, before the child process daemonizes. Many configuration
  errors which would previously have been reported in a logfile (after
  node startup), will now be signalled immediately, via stderr. In these
  cases, the "tahoe start" process will exit with a non-zero return
  code. (#2491)
* Unrecognized tahoe.cfg options are rejected at startup, not ignored
(#2809)
* ``tub.port`` can take multple (comma-separated) endpoints, to listen
  on multiple ports at the same time, useful for dual IPv4+IPv6
  servers. (#867)
* An empty ``tub.port`` means don't listen at all, which is appropriate
  for client-only nodes (#2816)
* A new setting, ``reveal-ip-address = false``, acts as a safety belt,
  causing an error to be thrown if any other setting might reveal the
  node's IP address (i.e. it requires Tor or I2P to be used, rather than
  direct TCP connections). This is set automatically by ``tahoe
  create-client --hide-ip``. (#1010)

Server-like nodes (Storage Servers and Introducers), created with
``tahoe create-node`` and ``tahoe create-introducer``, now accept new
arguments to control how they listen for connections, and how they
advertise themselves to other nodes. You can use ``--listen=tcp`` and
``--hostname=`` to choose a port automatically, or ``--listen=tor`` /
``--listen=i2p`` to use Tor/I2P hidden services instead. You can also
use ``--port=`` and ``--location=`` to explicitly control the listener
and the advertised location. (#2773, #2490)

The "stats-gatherer", used by enterprise deployments to collect runtime
statistics from a fleet of Tahoe storage servers, must now be assigned a
hostname, or location+port pair, at creation time. It will no longer
attempt to guess its location (with /sbin/ifconfig). The "tahoe
create-stats-gatherer" command requires either "--hostname=", or both
"--location=" and "--port". (#2773)

To keep your old stats-gatherers working, with their original FURL, you
must determine a suitable --location and --port, and write their values
into NODEDIR/location and NODEDIR/port, respectively. Or you could
simply rebuild it by re-running "tahoe create-stats-gatherer" with the
new arguments.

The stats gatherer now updates a JSON file named "stats.json", instead
of a Pickle named "stats.pickle". The munin plugins in
misc/operations_helpers/munin/ have been updated to match, and must be
re-installed and re-configured if you use munin.

Removed Features
----------------

The "key-generator" node type has been removed. This was a standalone
process that maintained a queue of RSA keys, and clients could offload
their key-generation work by adding "key_generator.furl=" in their
tahoe.cfg files, to create mutable files and directories faster. This
seemed important back in 2006, but these days computers are faster and
RSA key generation only takes about 90ms. This removes the "tahoe
create-key-generator" command. Any "key_generator.furl" settings in
tahoe.cfg will log a warning and are otherwise ignored. Attempts to
"tahoe start" a previously-generated key-generator node will result in
an error. (#2783)

Tahoe's HTTP Web-API (aka "the WAPI") had an endpoint named "/file/".
This has been deprecated, and applications should use "/named/"
instead. (#1903)

The little-used "manhole" debugging feature has been removed. This
allowed you to SSH or Telnet "into" a Tahoe node, providing an
interactive Read-Eval-Print-Loop (REPL) that executed inside the context
of the running process. (#2367)

The "tahoe debug trial" and "tahoe debug repl" CLI commands were
removed, as "tox" is now the preferred way to run tests. (#2735)

One of the "recent uploads and downloads" status pages was using a
Google-hosted API to draw a timing chart of the "mapupdate" operation.
This has been removed, both for privacy (to avoid revealing the
serverids to Google) and because the API was deprecated several years
ago. (#1942)

The "_appname.py" feature was removed. Early in Tahoe's history (at
AllMyData), this file allowed the "tahoe" executable to be given a
different name depending upon which Darcs patches were included in the
particular source tree (one for production, another for development,
etc). We haven't needed this for a long time, so it was removed. (#2754)

Other Changes
-------------

Documentation is now hosted at tahoe-lafs.readthedocs.io (not .org).

Tahoe's testing-only dependencies can now be installed by asking for the
[test] extra, so if you want to set up a virtualenv for testing, use
"pip install -e .[test]" instead just of "pip install -e ." . This
includes "tox", "coverage", "pyflakes", "mock", and all the Tor/I2P
extras. Most developer tooling (code-checks, documentation builds,
deprecation warnings, etc) have been moved from a Makefile into tox
environments. (#2776)

The "Welcome" (web) page now shows more detail about the introducer and
storage-server connections, including which connection handler is being
used (tcp/tor/i2p) and why specific connection hints failed to
connect. (#2818,
#2819)

The little-used "control port" now uses a separate (ephemeral) Tub. This
means the FURL changes each time the node is restarted, and it only
listens on the loopback (127.0.0.1) interface, on a random port. As the
control port is only used by some automated tests (check_memory,
check_speed), this shouldn't affect anyone. (#2794)

The slightly-more-used "log port" now also uses a separate (ephemeral)
Tub, with the same consequences. The lack of a stable (and
externally-reachable) logport.furl means it is no longer possible to use
``flogtool tail FURL`` against a distant Tahoe server, however
``flogtool tail .../nodedir/private/logport.furl`` still works just fine
(and is the more common use case anyways). We might bring back the
ability to configure the port and location of the logport in the future,
if there is sufficient demand, but for now it seems better to avoid the
complexity.

The default tahoe.cfg setting of ``web.static = public_html``, when
``NODEDIR/public_html/`` does not exist, no longer causes web browsers
to display a traceback which reveals somewhat-private information like
the value of NODEDIR, and the Python/OS versions in use. Instead it just
shows a plain 404 error. (#1720)