HTTP proxy support for node to node communication

[duck]

To use Tahoe-LAFS over the ​I2P anonymous network I have added HTTP proxy support to the Foolscap library. Most of the work is in Foolscap, but within Tahoe it must also be possible to enable this functionality and specify which HTTP proxy to use. All I2P nodes have this HTTP proxy enabled by default on 127.0.0.1:4444.

For the anonymous network use case, every connection to storage nodes and introducers will have to be through the anonymous network; it is not acceptable to mix anonymous and non-anonymous connections. The intention is to provide anonymity to both clients and node operators.

A reference implementation is available on ​http://duck.i2p.to/tahoe-lafs/ , as of today (2010-03-27) a test grid is operating inside I2P with 21 nodes, of which 6 storage nodes and 1 introducer.

Example of configuration in tahoe.cfg:

    [node]
    ...
    http_proxy = 127.0.0.1:4444

Snippit showing how this is used in node.py:

    http_proxy = self.get_config("node", "http_proxy", "")
    if http_proxy:
        self.tub.setOption("httpProxy", http_proxy)

[duck]

HTTP Proxy support to Foolscap

[duck]

Connected Foolscap tickets:

• ​#150 HTTP proxy support
• ​#151 Accept I2P destinations

[davidsarah]

Requires a test that we enable the foolscap option when the http_proxy line is present.

(The foolscap changes will also need additions to the foolscap test suite.)

[davidsarah]

Replying to davidsarah:

Requires a test that we enable the foolscap option when the http_proxy line is present.

This should probably go in source:src/allmydata/test/test_client.py​ , I think.

[zooko]

So the status of this ticket is that it is waiting for someone (ideally duck) to write tests, right? I guess that's what the "test" keyword means? Hey, let's make a new keyword: "test-needed". :-)

[zooko]

See also #510 (use plain HTTP for storage server protocol).

[zooko]

So back in comment:6 two months ago I set this ticket to "test-needed", and I haven't intended to do more on this ticket until duck (or someone) writes a test. But today I noticed that over on ​foolscap #150 and ​foolscap #151 duck has asked for the foolscap maintainer (Brian) to say whether he approves of the patches in principle and if so how to write a unit test for them. So the ball is back in our court. Brian: do you approve of these patches in principle?

[zooko]

Brian: please tell duck whether he has a chance of getting his patches accepted into foolscap trunk (assuming of course that the patches pass quality requirements -- unit tests, code review, docs, etc.). I think duck has been blocked by not wanting to invest effort into his patches when he hasn't received any indication from the foolscap maintainer on whether those patches could ever have a chance of inclusion.

[zooko]

Oh look! Brian updated ​foolscap #150 and ​foolscap #151!

[duck]

HTTP Proxy support to Foolscap v2

[duck]

The suggestion of Brian to name the foolscap property http-proxy instead of httpProxy has been taken.

In addition to this an unit test has been implemented as suggested by davidsarah in comment:4. test_node.py seemed to be a better place than test_client.py (comment:5) as this option applies to the [node] section and changes are made to node.py.

Please review version 2 of the patch.

[davidsarah]

Given Brian's ​comment:3 on foolscap#150 that the proxy is actually relaying foolscap (which only initially looks a little like HTTP), I think the option name should not have "http" in it. Technically it is a "storage connection proxy", although that is a little long.

Another reason not to use "http" is that that could be confused with an HTTP proxy for the web-API (which we intentionally do not support).

[duck]

Which name would you approve of for the "introducer and storage connection proxy that speaks a little HTTP proxy and is used by foolscap to make any outbound connections" configuration option?

• foolscap-proxy
• outbound-proxy
• i2p-proxy
• tub.proxy

[zooko]

Assigning to davidsarah to answer duck's question from comment:18. (Brian may also have an opinion.)

[davidsarah]

outbound-proxy for the config option. It's not specific to i2p, so that rules out i2p-proxy. The existing tub. options control the tub for this storage server. And outbound-proxy is more specific than foolscap-proxy.

[davidsarah]

Replying to davidsarah:

outbound-proxy for the config option.

Make that outbound_proxy. Other config options use _ instead of -.

[zooko]

Removing the review-needed tag until someone (perhaps duck) updates the name of the config option.

[killyourtv]

outbound-proxy

[killyourtv]

Attached is an updated patch (0001-outbound-proxy-support.patch​ ) that changes the config option to outbound-proxy. The patch applies cleanly to current trunk.

(I'm still trying to learn Python so please forgive any n00b errors. I also need to learn how to write unit tests).

[daira]

BTW, the Version field is intended to reflect the version in which an issue was first reported, so it's not necessary to update it unless it was originally set incorrectly.

[zooko]

Jeff "psi" "ampernand" and I have been looking at this, and we don't think HTTP-proxying is the best way to do accomplish this. Because HTTP-proxying is meant to provide a request-response style, but Foolscap needs more of a "bidirectional byte-stream" style.

A potentially better way to accomplish it, which we are now poking at, is Twisted Endpoints -- see ​http://foolscap.lothar.com/trac/ticket/203 . One potential advantage of that — if it can be made to work — is that it might make it easier to support Tor, cjdns, SSL/TLS, IPv6, and maybe other cool networking protocols.

[daira]

We agreed in today's Nuts & Bolts meeting to bump better Tor/I2P support out to 1.11.0.

[warner]

Milestone renamed

[warner]

moving most tickets from 1.12 to 1.13 so we can release 1.12 with magic-folders

[exarkun]

Moving open issues out of closed milestones.

[meejah]

Ticket retargeted after milestone closed