UPnP

[zooko]

I would like it if the Tahoe node would talk to a UPnP-enabled firewall and automatically arrange for port-forwarding as needed.

[zooko]

Here is a free-software UPnP implementation:

​https://coherence.beebits.net/wiki

But it doesn't have the internet-gateway-client functionality yet.

Frank Sholz, the author of Coherence, writes:

for the port-forwarding we need to add a internet-gateway-client	
http://upnp.org/standardizeddcps/igd.asp				
it is in the file WANIPConnection					
it looks not difficult						
we need something like this
      https://coherence.beebits.net/browser/trunk/coherence/upnp/devices/media_server_client.py
but with the idg services
but that's then already the device, won't be only afew more lines of python code   
https://coherence.beebits.net/browser/trunk/coherence/upnp/services/clients/av_transport_client.py
and that's the second thing					
a simple mapping of python methods to the soap calls

[ghazel]

Unrelated to the Coherence route, I've had good luck with a python-wrapped version of MiniUPnP, ​http://miniupnp.free.fr/ If all you're looking for is port mapping, it's probably the smallest way out. If you plan to provide data to set-top boxes and the like, Coherence looks like it is definitely a better way to go. Adding IGD support will get you basic compatability, but there are a lot of ugly parts of dealing with improper implementations on consumer routers. Just my 2c.

[zooko]

I was just trying to make my storage servers for the volunteer grid more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in.

So this is just a note to remind myself that I wish someone would implement this. :-)

[swillden]

Replying to zooko:

I was just trying to make my storage servers for the volunteer grid more reachable by setting up IP forwarding rules, and I realized that this would be nice and easy -- indeed would have happened automatically without my thinking about it -- if we had UPnP built in.

So this is just a note to remind myself that I wish someone would implement this. :-)

I've fiddled a little bit with miniupnpc, which ghazel already mentioned. It's a C library but includes a Python wrapper. It's pretty simple. The following code was all it took to tell my gateway (a Linux box running linux-igd) to set up a forwarding for my Tahoe node:

import miniupnpc

u = miniupnpc.UPnP()
u.discoverdelay = 200
u.discover()
u.selectigd()

external_port = 3456
r = u.getspecificportmapping(external_port, 'TCP')
while r != None and external_port < 65536:
    external_port += 1
    r = u.getspecificportmapping(external_port, 'TCP')
u.addportmapping(external_port, 'TCP', u.lanaddr, 3456, "Tahoe %u" % external_port, '')

Obviously, that needs a lot of error checking, but it's still really straightforward. I ran that script from the command line, and watched the log on my gateway to verify that the mapping was set up. I then verified that I can connect to my local Tahoe server from outside. Works fine.

[zooko]

Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-)

[swillden]

Replying to zooko:

Cool! Thanks for reporting this! Now we need someone to write tests for it and integrate the miniupnpc dependency into our packaging system. :-)

Writing tests for it might be challenging, since good tests would require the presence of a UPnP IGD to respond to the registration/de-registration, which would require providing a rudimentary (at least) IGD daemon in the code base just for testing.

[warner]

does anyone know what percentage of home users have functional UPnP firewalls? I seem to remember reading somebody's report on the p2p-hackers mailing list that suggested that a large fraction (half?) of the boxes which claimed to support UPnP were in fact completely broken.

[zooko]

​http://zgp.org/pipermail/p2p-hackers/2006-June/003880.html

Alex Pankratov gave that 50% number on p2p-hackers in 2006. Around the same time allmydata.com implemented UPnP in their proprietary product (the one that preceded Tahoe) and measured the success rate among allmydata.com customers. Unfortunately I don't have or remember the numbers. We should ask Jim McCoy? and Rob Kinninmont if they have or remember the results.

I assume that deployment and robustness of UPnP has improved since then. ;-)

Because of foolscap's nice design of using multiple connection hints in parallel and requiring strong SSL authentication, a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right?

[warner]

I assume that deployment and robustness of UPnP has improved since then. ;-)

Thanks for the reference. I admire your confidence :)

a broken UPnP implementation suggesting a wrong IP address and port number wouldn't do any harm, right?

correct. Worst case, the node thinks it is externally visible but is not.

Implementation-wise, we probably need some improvements in the overall IntroducerClient behavior, since the current code assumes that it can get its location hints quickly at startup, and that they never change after that. We need something to handle changes to the location hints at runtime (including the initial determination, which usually requires running /sbin/ifconfig to compute and is thus asynchronous) and then send a new announcement to the Introducer each time. Foolscap is not entirely happy with changing location hints either: FURLs are computed once at startup (specifically at tub.registerReference time), and not updated afterwards, so we'd need some more work to make that behave smoothly.

We should probably implement #445 (relay) and #50 (STUNT) eventually, too.

[zooko]

Thanks for the reference. I admire your confidence :)

Waaataminute, by "admire" you mean "mock", don't you?

As to the testing question that Shawn raised, I think we could write a fake IGD daemon in the test code, for example one which just returns a canned answer in response to the UPnP "are you there" and another canned answer in response to the UPnP "can I have an external port please?", and so on for all the UPnP IGD client functionality that we need. That is slightly complicated (especially if the UPnP IGD client we are using uses irreproducible randomness in its queries and expects the server to echo that randomness), but probably not trickier than the other "network server fakery" that is already in the TahoeLAFS unit tests.

[zooko]

Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as miniupnp, right? I'm not sure about this.

[swillden]

Replying to zooko:

Hrm... on second thought, that sort of testing belongs in the tests for the IGD client package, such as miniupnp, right?

Makes sense to me. If it were necessary to write test cases to validate the correct functioning of every library used, you'd never have time to write any code.

[zooko]

If you like this ticket, you may also like #50 (STUNT/ICE), #169 (tcp hole-punching!), #445 (implement relay: allow storage servers behind NAT), and #754 (merge manually specified tub location with autodetected tub location).

[jrydberg]

A simple UPNP library, that also aims at supporting NAT-PMP: ​http://github.com/jrydberg/natmap

[USSJoin]

Just another vote (over the *three years* this ticket has been outstanding) for this. Additional use case, that I know applies to Zooko (as well as many other users): laptops that move from home to work should ideally tell the firewall in each place to let connections through, each time they move. UPnP would make this immediate and automatic. (Just run it on startup, naturally-- not just a one-time configuration.)

[zooko]

This came up on the mailing list again, as it does every year, and this time I found the CoralCDN stats:

​http://tahoe-lafs.org/pipermail/tahoe-dev/2010-June/004469.html

CoralCDN did some large-scale stats gathering: ​http://illuminati.coralcdn.org/stats/ . They say that about 75% of the kind of people who participate in CoralCDN surveys are behind NAT. There's no timestamp on that page, but judging from the late of their last publication I guess it was 2007. They also did some measurements of how much added latency you would incur by going through a relay, which is something that you might want to take a look at since you've been hacking on foolscap relay.

[zooko]

jrydberg's ​http://github.com/jrydberg/natmap looks good. A possible next-step for this ticket would be to add a dependency on natmap and invoke natmap some start-up code, possibly from where-ever iputil gets invoked to determine your current IP address(es).