Foolscap-0.12.0 released

[Brian Warner]

Heads up, I just shipped foolscap-0.12.0 to pypi. Foolscap is a secure
RPC protocol for Twisted, which Tahoe uses for connections to storage
servers.

This release deprecates a number of the automatic configuration tools
which we no longer think are a good idea (TCP port allocation and
automatic IP-address detection). Applications are encouraged to give
users way to specify an externally-visible hostname (or IP address) for
server programs. (Tahoe-LAFS trunk stopped using these features a few
months ago, in anticipation of this change).

I had wanted to put off this release until we had the Tor/I2P/SOCKS
connection handlers in place, and had replaced the use of pickles in
flogfiles with JSON, but there was an urgent deadline. The unit test
suite in Foolscap-0.11.0 was making network connections (to exercise the
automatic IP-address detection), which violated debian policy for
reproducible builds, and Tahoe was slated for removal from the "testing"
distribution because it's dependency fails to build from source. I'm
hoping to get an even newer Foolscap release (0.13.0) out in the next
few weeks with the Tor stuff.

The full changelog (NEWS) file is attached below.

cheers,
 -Brian




* Release 0.12.0 (20-Jul-2016)

** API changes: no more automatic configuration

Foolscap has moved from automatic listener configuration
(randomly-allocated TCP ports, automatically-determined IP address) to
using more predictable manual configuration. In our experience, the
automatic configuration only worked on hosts which had external IP
addresses, which (sadly) is not the case for most computers attached to
the modern internet. #252

Applications must now explicitly provide Foolscap with port numbers (for
Tub.listenOn) and hostnames (for Tub.setLocation). Applications are
encouraged to give users configuration controls to teach Foolscap what
hostname and port number it should advertise to external hosts in the
FURLs it creates. See https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2773
for ideas.

The specific API changes were:

- Tub.setLocationAutomatically() has been deprecated
- Listener.getPortnum() has been deprecated
- calling Tub.listenOn("tcp:0") is also deprecated: callers should
  allocate a port themselves (the foolscap.util.allocate_tcp_port
  utility function, which does not block, has been added for this
  purpose).

Foolscap tools like "flappserver create" and "flogtool create-gatherer"
will no longer try to deduce their external IP address in an attempt to
build externally-reachable FURLs, and will no longer accept "tcp:0" as a
listening port (they now default to specific port numbers). Instead,
they have --location= and --port arguments. The user must provide
'--location' with a connection-hint string like
'tcp:hostname.example.org:3117' (which is put into the server's FURLs).
This must match the corresponding '--port' argument, if provided.

- for all tools, if '--port' is provided, it must not be tcp:0
- 'flappserver create' now requires --location, and '--port' defaults to
  tcp:3116
- 'flogtool create-gatherer' requires --location, default is tcp:3117
- 'flogtool create-incident-gatherer' does too, default is tcp:3118

For backwards-compatibility, old flappservers will have "tcp:0" written
into their "BASEDIR/port" file, and an empty string in
"BASEDIR/location": these must then be edited to allow the flappserver
to start. For example, write "tcp:12345" into "BASEDIR/port" to assign a
portnumber, and "tcp:HOSTNAME:12345" into "BASEDIR/location" to expose
it in the generated FURL.

** Other API changes

Tub.listenOn() now takes a string or an Endpoint (something that
implements twisted.internet.interfaces.IStreamServerEndpoint). This
makes it possible to listen on non-IPv4 sockets (e.g. IPv6-only sockets,
or unix-domain sockets, or more exotic endpoints), as long as
Tub.setLocation() is set to something which the other end's connection
handlers can deal with. #203 #243

The "DefaultTCP" handler (which manages normal "tcp:HOST:PORT"
connection hints) has been moved to foolscap.connections.tcp . This
makes room for new Tor/I2P/SOCKS handlers to live in e.g.
foolscap.connections.tor . #260

Connection handlers are now allowed to return a Deferred from
hint_to_endpoint(), which should make some handlers easier to write.
#262

Note that RemoteReference.notifyOnDisconnect() will be deprecated in the
next release (once all internal uses have been removed from Foolscap).
Applications should stop using it as soon as possible. #42 #140 #207

** Compatibility Changes

This release removes support for the old (py2.4) "sets" module. This was
retained to support applications which were trying to maintain py2.4
compatibility, but it's been so long since this was necessary, it's time
to remove it.

** Other Changes

The internal `allocate_tcp_port()` function was fixed: unexpected kernel
behavior meant that sometimes it would return a port that was actually
in use. This caused unit tests to fail randomly about 5% of the time.
#258

IPv6 support is nearly complete: listening on a plain TCP port will
typically accept connections via both IPv4 and IPv6, and the DefaultTCP
handler will do a hostname lookup that can use both A and AAAA records.
So as long as your server has a DNS entry that points at its IPv6
address, and you provide the hostname to Tub.setLocation(), Foolscap
will connect over IPv6. There is one piece missing for complete support:
the DefaultTCP connection handler must be modified to accept
square-bracketed numeric IPv6 addresses, for rare situations where the
host has a known (stable) IPv6 address, but no DNS name.