use ipv6

[warner]

Shawn Willden points out that IPv6 is a likely way to deal with the servers-behind-NAT problem (#169, #445, #49, #50), since a lot of ISPs offer v6 tunnel services these days, which effectively gives the server host a publically-reachable v6 address.

To make this work, we need a couple of things, probably in the following order:

• python must handle v6 addresses, in the socket module
• twisted must handle v6 addresses: connecting to them, reporting them when asking a socket who it is connected to, and enumerating them on listenable interfaces. The existing IPv4Address class must grow a corresponding IPv6Address partner.
• foolscap must handle them: putting them in connection hints (​foolscap#60 has some notes on formats), connecting to them, etc
• tahoe must handle them: formatting them into Welcome page listings, passing them through the Introducer without problems, logging them correctly

Let's collect pointers to other tickets (especially the twisted ones) and progress reports here.

[warner]

​twisted#3014 looks to be the main v6-in-Twisted ticket, and is moving very slowly (created two years ago). It looks like there's a bunch of v6 address lookup code already in place, but perhaps the actual make-a-TCP-connection code is not.

[warner]

​Foolscap#155 is the new v6-in-Foolscap ticket. Doesn't look too hard once the twisted ticket is done. That twisted ticket is still open, although I think there's been some good progress on it in the last month.

[davidsarah]

The relevant Twisted ticket now seems to be ​http://twistedmatrix.com/trac/ticket/5085, which is fixed.

[ClashTheBunny]

With progress by Marcus Wanner on ​Foolscap#155 I felt some motivation to work on IPv6 inside of Tahoe. Right now I've just done some of the ip_utils that are specific to IPv4 including some test cases that were missing. If my IPv6 work never makes it into Tahoe, my test cases would be useful to pull in.

You can follow my progress at and send pull requests to ​github. It's nothing much right now, but it's where I'm putting my work, so if anybody has anything to add, send it my way.

I asked on IRC and nobody else seems to be working on this, so let me know if you have something already done.

There may be some point to document a way to tunnel IPv4 over IPv6 until this work is done so that people can take advantage of IPv6 without the platform support within Tahoe. Something as simple as nc and some pipes.

[davidsarah]

Thanks! I don't think anyone else is working on this at the Tahoe end, so feel free to continue.

[davidsarah]

Screenshot from 2013-02-14

[davidsarah]

ClashTheBunny? wrote on tahoe-dev:

I've made some progress with IPv6. The first time a node is brought up it listens on both IPv4 and IPv6. I still have to figure out many side cases and test what is best when there is only IPv6, only IPv4, or both. It also needs to set the port correctly in the furl file after it is first run so that it listens to the correct protocol on the next runs. It's still communicating over IPv6!

Yay!

Re: the screenshot​, the "Connected?" column should probably show IPv6 addresses in the same format they appear in furls.

After these side cases are thought out, tests, tests, and more tests!

Excellent.

[ClashTheBunny]

How time flies. I've made some good progress on the IPv6 implementation and now need some good advice as to what to write new tests for. I could also use some code review. Everything implemented as far as I can tell:

• I've added a new boolean option "ipv6privacy" that will turn off all addresses that contain a MAC address
• The fe80::/10 addresses are silently dropped
• I've added a new boolean option "preferipv4" that will list address in an order that should prioritize IPv4 addresses when true. It will also listen on IPv4 interfaces only. I am pretty sure that this works, but it would be nice for some people to do some tests. I think this may need to be converted to a tri-state of ipversion = v4, ipversion = preferv4, and ipversion = preferv6. Suggestions welcome.
• IPv6 and IPv4 addresses are now listed in the "Connected?" column as they appear in FURLs.
• I detect if I'm listening on IPv6, and write my port file as an IPv6 port in that case. If IPv6 comes up the first time, may as well make it come up always. If it doesn't come up, you need to delete the port file and bring it up again.
• I've written a few tests, and updated others that needed updating, but as I said above, I would like some advice as to what else needs to be tested, and what should be assumed from foolscap's tests passing.

Things not complete that may be good to include in this:

• The webserver still is IPv4 only. It seems to be a larger update than this one because more of the low level stuff is done there and not handled by foolscap like here.
• I don't know exactly where logging of IP addresses happen, but I haven't seen any addresses logged yet. I've grepped around the code and didn't see anything, but a pointer to where this would happen would be nice.

Things required:

• foolscap from Marcus Wanner's ipv6 branch on github: git clone ​https://github.com/marcuswanner/foolscap -b ipv6
• Twisted 12.1
• My ipv6 Tahoe branch.
• IPv6 of course!

Remember, there are probably still bugs, so treat this as dangerous with data. I don't think I touched anything mission critical though, and all the old tests pass, so I wouldn't be too scared either.

[ClashTheBunny]

I converted preferv6 to a tri-state of ipversion = v4, ipversion = preferv4, and ipversion = preferv6. In a 7 node network with two nodes only reachable via IPv6, the upload succeeds on a "preferv4" host and a "preferv6" host, but fails on the "v4" host. So I know that v4 works as expected. I'm happy with the way preferv6 and preferv4 work. They advertise the right version first, but still work with either version.

I need to get some real testing done with Windows before 7 or another operating system that has a separate ipv4 stack from an ipv6 stack. If anybody is willing to try this, I would greatly appreciate it.

[gdt]

A few comments:

• the URL of your git repo/branch should probably be in this ticket
• Probably someone (you?) should file a separate ticket for making the WUI listen on ::1 (and presumably then the CLI interface could use that next, another ticket).
• whether a node is willing to use v4 vs v6 for outgoing connections should be separate from whether the node listens on v4 or v6 (foolscap)
• the concept of the port file seems a little off. nodes listen on an address/port pair, or rather a set of them. what port means now is "put this together with INADDR_ANY". But, it could be "put it with INADDR_ANY, and also with INADDR6_ANY". So probably the node config needs a config to list the address families on which to listen, defaulting to all OS-supported families. (For extra credit, test on a BSD system without INET in the kernel - I need to see if that still builds, and if not fix it.) I suggest "IncomingFamiles? = inet6, inet", being a set.
• for outgoing connections, the introducer furl, or data from the introducer, will have a list of addresses. I think it makes sense to 1) include a list of acceptable address families, defaulting to all OS-supported families and 2) have a way to express preference. For preference, I would start with v6, following the modern tradition. So I can see this being a "OutgoingFamilies? = inet6, inet" and it being a list rather than a set.

[ClashTheBunny]

Replying to gdt:

A few comments:

• the URL of your git repo/branch should probably be in this ticket

It's listed in my first comment, but it's here (for easier grep-ability): ​https://github.com/ClashTheBunny/tahoe-lafs/commits/ipv6

• Probably someone (you?) should file a separate ticket for making the WUI listen on ::1 (and presumably then the CLI interface could use that next, another ticket).

Yeah, I'm thinking that either we should make three tickets with this as the parent, or create two more tickets with those parts of the process. If we did three tickets, I would call mine "Tahoe-LAFS transport over IPv6" or something like that. I'll let zooko or David-Sarah figure that out, because they know more about the management of the project.

• whether a node is willing to use v4 vs v6 for outgoing connections should be separate from whether the node listens on v4 or v6 (foolscap)

Agreed. I'll look into how that works in foolscap. The problem is that there aren't many controls that I have. Right now I can specify which addresses are advertised and in which order. I can also specify if the listen port is TCPv4 or TCPv6. I don't think I can tell foolscap not to make connections to a certain address family, but as I said, I'll look into that.

• the concept of the port file seems a little off. nodes listen on an address/port pair, or rather a set of them. what port means now is "put this together with INADDR_ANY". But, it could be "put it with INADDR_ANY, and also with INADDR6_ANY". So probably the node config needs a config to list the address families on which to listen, defaulting to all OS-supported families. (For extra credit, test on a BSD system without INET in the kernel - I need to see if that still builds, and if not fix it.) I suggest "IncomingFamiles? = inet6, inet", being a set.

I don't know if the first case exists. I think I can specify INADDR_ANY and INADDR6_ANY. Linux automatically makes INADDR6_ANY also receive IPv4 connections if /proc/sys/net/ipv6/bindv6only is set to the default value of 0. In NetBSD I think this would be sysctl -w net.inet6.ip6.v6only=0. This INADDR_ANY and INADDR6_ANY translate to this in a tub entry:

0 or tcp:0 is any free port the OS will give us on only INADDR_ANY tcp6:0 is any free port the OS will give us on INADDR6_ANY, if your OS listens on INADDR6_ANY for INADDR_ANY connections also, you get both. This is Linux's default, I think every BSD, and I've heard that Windows 7 also, but Windows XP/Vista have seperate V6 and V4 stacks, so it only listens on INADDR6_ANY.

<any number> or tcp:<any number> listens on INADDR_ANY on that specific port tcp6:<any number> listens on INADDR6_ANY on that specific port, with the above caveats about stacks and listening on both.

Either way, I'll set up bindv6only on a test machine I'm running to make sure that the expected behavior works (or that ipv4 connections don't work!).

• for outgoing connections, the introducer furl, or data from the introducer, will have a list of addresses. I think it makes sense to 1) include a list of acceptable address families, defaulting to all OS-supported families and 2) have a way to express preference. For preference, I would start with v6, following the modern tradition. So I can see this being a "OutgoingFamilies? = inet6, inet" and it being a list rather than a set.

This is essentially what I did with ipversion above. I don't know if there is a real way of asking foolscap to actually prefer one over the other, but I do know that if I list IPv6 addresses at the end of the list of addresses advertised to the introducer, I, more often than not, get IPv6 connections. The opposite is also true, if I list them first, I rarely get IPv6 connections. If you look at the code that includes ipversion it's kinda a combination of these last two comments. I now understand that they can be split into two variables:

IncomingFamily? = inet or inet6, implementation details would then state that, depending on your OS and settings this will give you v4, v6, or both.

and PreferredOutgoingFamily? would do the previous reordering of addresses when advertising to the introducer my addresses.

I don't think I can tell the introducer that I support one type of address or another, but that would have to happen in foolscap. Also, it would be nice to file a bug against foolscap to have a true preference of family be possible. The bug tracker is currently down, so I keep making comments on Marcus Wanner's latest commit to his ipv6 branch of foolscap on github (​https://github.com/marcuswanner/foolscap/commit/d890c7739008103ebdd2ecdb5dab971ef9cee484). Right now I'm going to split my tri-state variable 'ipversion' into the two you suggested. We'll hope that foolscap is able to either be fixed or keeps it current behavior. I'll also make a comment in the code about how it works for me but there is no official way of doing this yet.

[ClashTheBunny]

Greg Troxel helped me understand some things about dual stack vs split stack v4/v6 implementations, it should now work in both cases. The configuration options are now outgoingfamily as either preferv6 or preferv4 and incomingfamily as inet, inet6, or both.

I still don't think that I can stop foolscap from connecting to IPv6 addresses that are advertised to it if I have IPv6 connectivity to that node. Outgoingfamily still chooses to advertise IPv6 addresses first or last based off of my experience with foolscap, but it still doesn't prevent connections over IPv6 or IPv4.

I just pushed these changes to github, so keep testing away everyone!

[gdt]

I forgot about foolscap's try-them-all-first-one-wins plan. Given that, I don't think it makes sense to fight the foolscap way. The usual reasons to try v6 first are to promote v6 adoption and to avoid NAT pain. The reasons to configure v4 first instead are because v6 is flaky or slow (due to infrastructure/routing, not protocol).

Longer term, foolscap should become aware of AF preference and support that somehow.

[davidsarah]

Given foolscap's try-them-all-first-one-wins behaviour, is there any real need for the preferv4 option? If the required v6 connectivity isn't available, then the attempted v6 connections should fail quickly (or am I wrong about that?) So the only reason to use preferv4 would be if the v6 connections can succeed but are not reliable in some way. But, in that case it would still be possible for a v6 connection to be made if preferv4 is set, and so setting that wouldn't solve the unreliability.

[gdt]

In a multi-AF world, foolscap not having a "only use this AF" runtime config is at least a misfeature if not a bug. But until foolscap gets that, I don't see the point in sorting.

[ClashTheBunny]

Updated branch to include tests for IPv6 in client tests and introducer tests.

Added option to include fe80::/10 addresses but is off by default and should not be documented. In the future, it will allow more flexibility with fewer changes. If somebody really wants an fe80 address to show up, they can read the source code and enable the option.

An option for end to end testing would be to turn off IPv4 in all of the regular tests. This would allow all of the other tests to be tested over IPv6, thereby testing IPv6.

All my new tests pass, including the tests that I added for IPv4 addresses.

[davidsarah]

Replying to ClashTheBunny:

Added option to include fe80::/10 addresses but is off by default and should not be documented. In the future, it will allow more flexibility with fewer changes. If somebody really wants an fe80 address to show up, they can read the source code and enable the option.

I think we prefer all options to be documented.

[gdt]

Replying to davidsarah:

Replying to ClashTheBunny:

Added option to include fe80::/10 addresses but is off by default and should not be documented. In the future, it will allow more flexibility with fewer changes. If somebody really wants an fe80 address to show up, they can read the source code and enable the option.

I think we prefer all options to be documented.

I think the point here is that including fe80::/10 is basically broken but there are perhaps reasons why someone would want to experiment. I think clashthebunny is talking about a variable to set in the sources, not command-line support. I would say that if there's a requirement to expose this in command-line and docs, then the option shouldn't exist. I haven't seen a valid use-case for it yet.