synchronous node startup

[dawuud]

Synchronous node startup is part of the native Tor integration as specified in #517

[dawuud]

this untested rough draft attempt at synchronous startup for tahoe in my dev branch: ​https://github.com/david415/tahoe-lafs/tree/2491.synchronous-node-startup.0

this change uses get_all_ip_addresses from the new foolscap helper function specified in this trac ticket: ​http://foolscap.lothar.com/trac/ticket/238#comment:1

[warner]

More details copied from #517:

At node-creation time (e.g. tahoe create-node):

• tahoe create-node should allocate (or be told) a port to listen on, and write it into tahoe.cfg tub.port as a server endpoint descriptor string (so tcp:12345, or tcp:12345:interface=127.0.0.1).
• it should also detect (or be told) an IP address to use in the FURL, and write it as tub.location (as a Foolscap connection hint, so tcp:HOST:PORT)

At runtime (e.g. tahoe start):

• Try to get the port number from BASEDIR/client.port (which must not be "0"). If that's empty, read tahoe.cfg tub.port (which might be missing, which means "AUTO"; we can probably reject "0" here too).
• if our port is "AUTO" (which only happens if we have a node created before the above tahoe create-node was deployed, but which has never been started with tahoe start), then we use the utility function to allocate a port, and we write it into BASEDIR/client.port.
• Now we know what port to listen on. Build a server endpoint descriptor string and pass it to tub.listenOn().
• Try to get the tub location from tahoe.cfg tub.location.
• If that's empty, use the synchronous non-Twisted IP-address detection function to get a list of IP addresses. Combine that with the port we're listening on (from above) to build a list of hints. Join them with commas.
• Pass the hints into tub.setLocation().
• Now call the rest of the Node/Client startup functions.

Other code changes/accomplishments:

• node.Node._tub_ready_observer and when_tub_ready go away
• several client.Client methods stop using when_tub_ready (and stop using Deferreds): init_introducer_client, init_storage, init_control, init_helper, init_key_gen.
• in particular, we call tub.setLocation() *before* calling any of those init methods, so they can safely use tub.registerReference

Other tasks necessary to make this work:

• move the controlport and logport to separate Tubs, with a separate (ephemeral) key, listening on a runtime-allocated port, and 127.0.0.1
• consider adding a config knob to override this: to set a stable port and key, so you can copy the logport.furl to a separate machine and use flogtool tail after a reboot.

[warner]

I've pushed warner/2491-sync to my github, which rebases and builds upon your branch. It passes most tests, but a couple are still broken:

• test_system leaves a dirty reactor
• test_introducer.ClientSeqnums is broken.. probably shallow

I like how much simpler the setup code is without needing to wait for the Tub to be ready.

[warner]

(so far, my branch addresses the "runtime" changes above, and the "other code changes/accomplishments", but doesn't change create-node, and doesn't touch controlport/logport)

[warner]

I've split out the tahoe create-node --location= UI task to #2773.

[Brian Warner <warner@…>]

In 6dfcd52/trunk:

iputil.py: copy allocate_tcp_port() from foolscap

refs ticket:2491

[Brian Warner <warner@…>]

In e9e1265/trunk:

SystemTestMixin?: don't auto-detect IP addresses

The tubs created for test_system should always use 127.0.0.1 .

refs ticket:2491

[warner]

​https://github.com/tahoe-lafs/tahoe-lafs/pull/270 is ready for some review.

[Brian Warner <warner@…>]

In 96c6259/trunk:

Teach tub to start synchronously even with ip autodetect

• remove when_tub_ready() from all code
• synchronous-ify all node/client/introducer startup code

refs ticket:2491

[warner]

Ok, that's merged, thanks meejah for the +1.

Still left to do for this ticket:

• move controlport/logport to a separate Tub

And we should open a separate ticket for the changes to tahoe create-node.

[warner]

This fixes #726, where port-already-in-use errors were written to the log instead of being signalled immediately.

[warner]

#2773 is for the tahoe create-node argv changes: adding --hostname / --location / -- port

[warner]

As mentioned in #1377, ​http://twistedmatrix.com/trac/ticket/823 might help address one of the problems this is addressing: errors at startup getting written to the logfile instead of the controlling terminal (and the tahoe start process reporting success, not failure). twistd basically now does the old buildbot trick of waiting for the new child process to get running (by reading from a pipe-to-self) and reporting a short error message if it fails.

[warner]

#2794 is for moving the controlport/logport to a separate Tub, so we can close this ticket now