[tahoe-dev] [tahoe-lafs] #867: use ipv6

[Randall Mason]

Greg,

I'm sorry it's taken so long to respond.  This has been one of the most in
depth emails that I've read from a subject matter expert.  My list of read
RFCs is quite a bit longer at this moment.  I'm sorry that I'm implementing
this patch and not somebody else who more thoroughly knows what IPv6 is
from a theoretical point as apposed to the place where I'm approaching it
from a user point of view.  I'm also a pragmatist in some senses and may be
oversimplifying things because I haven't done my homework on how long
actual delays are, what different implementations exist and how these two
things will impact performance.

I really appreciate somebody who knows more about IPv6 than what Hurricane
Electric will certify as a "sage".  A fool could follow a blog for a half a
day to get there and who knows how much better I am than that.  I hope that
you take my questions as something like a student-teacher questioning.  As
I said above, your email has pushed me to learn more about this complicated
web of RFCs that make up IPv6, which has been a very fulfilling journey.

Here's what I've come up with:

On Sat, Feb 16, 2013 at 4:19 PM, Greg Troxel <gdt at ir.bbn.com> wrote:

>
>   Would you be able to elaborate about this?  Specifically about my use
> case
>   of two hosts on tunnel brokers, but link-local.  I feel it's important,
> and
>   nobody's going to be typing in the furls manually, so who does it benefit
>   to have less capability than more?
>
> It's not about the benefit of less capability.  It's about avoiding
> mess.  If the fe80:: addresses of a host are advertised to an
> introducer, then almost all nodes (in general) will not be able to reach
> that address.  Even worse, it will time out rather than get a connection
> refused.  And it leaks the MAC address, which its itself a security
> concern (possible with stateless autoconf, but people who have
> configured static addresses to avoid this should not be exposed).
>

Avoiding mess is always good.  What happens currently with
169.254.0.0/16addresses in Tahoe?  What about RFC1918 addresses?  What
about
127.0.0.0/8? Are they deprioritized so that connections happen to them in
the last case?  What is the delay if the connection times out?  Does Tahoe
only connect in serial, as apposed to starting to open up x different
connections and take the first one that connects?  Does Tahoe use the order
of the address in the furl?  What's the current algorithm for IPv4
addresses and the justification for it?

In the v6 specifications, the intent of link-local addresses (in
> fe80::/16) is that they are only used for things like neighbor discovery
> and routing protocols.


This is true, but in [RFC4291](http://tools.ietf.org/html/rfc4291#page-11)
Link-Local is for three cases, the two sited, and when no routers are
present:

   Link-Local addresses are designed to be used for addressing on a
   single link for purposes such as automatic address configuration,
   neighbor discovery, or when no routers are present.

It is not a requirement that Link-Local addresses are used, but it is one
of the purposes of Link-Local addresses.  My use only falls in the third
category, but I think that may be enough.

 When you use a link-local address, the address
> itself does not specify to the host stack which interface to use.
> That's why you have the %wm0 or whatever showing up in the display
> representation, which is based on the ifindex being inserted in one of
> the bytes of the address.  To have the address be used by another host,
> it has to not have that, and has to insert its own index.  So passing
> them around is non-obvious; typically a routing protocol will just
> record the other side's address and reuse it - but there it has its
> *own* ifindex, specifying the interface the packet arrived on, rather
> than tha remote side's index.
>

After a lot of research I get what you're saying.  I do filter off the
ifindex that BSD includes in the ifconfig -a results so it looked familiar,
but I didn't know how necessary it was on Linux.  Picturing a multihomed
computer, say WiFi + Ethernet, there is no way to know WHICH fe80::
interface we should use.  Why is it that BSD and Windows don't seem to have
a problem.  I ping6 fe80 addresses without an interface identifier on those
two platforms and don't have a problem.  Even with multihomed devices.  Is
this a deviation from the standards with Linux or some bizarre speed
optimization?  Why can't it ARP all interfaces?


> tahoe is designed primarily as a wide-area protocol; in the general case
> all nodes are expected to be able to open a TCP connection to the
> introducer, and then all clients are expected to open a connection to
> all servers.  To do this sensibly, the introducer and each server node
> need a globally-routable address.
>

This is the main purpose for my work.  I think that the global nature of
IPv6 and Tahoe are a match made in heaven.  It will be a wonderful day when
I can have a Mobile IPv6 my laptop and my phone and never think about which
link to communicate over, multiple DNS entries for a single host, what
network it's hooked up to, and which interface is actually connected right
now.  At least right now it's light years ahead of NAT and port forwarding.
:-)


> I should point out that my bias is that running tahoe only locally
> doesn't make sense in the first place.  There are other filesystems that
> deal with resilience against disk failure and some of them have better
> performance.  The real point is to get resilience against hazards that
> affect an entire site, so wide-area connectivity is IMHO intrinsically
> tied to the main use case.
>
This exactly.  If somebody thinks this is a replacement for ceph or RAID,
they will be sorely disappointed with the speed and how it is a file store
and not a file system.

>
>   Other advantages are that they are not routed, so that they can be more
>   "secret" than other addresses.  If you didn't want the world to know that
>   you were using Tahoe, preferring more local over more remote addresses
>   could be better.
>
> I think this is the usual security-through-obscurity issue, and I don't
> think it makes sense.  Outsiders can no more tell that you are using
> tahoe with global addresses on your ethernet than they can tell about
> private addresses.
>

True, but this isn't about security through obscurity, it's about security
and while we're at it, why not obscurity as well.  There is nothing wrong
with obscurity, until people think that it can replace security.  Security
and obscurity is even better than just security.  For example I would never
send unencrypted information about dissidents in an oppressive regime, but
I'd be happy if I didn't have to spend the rest of my life in jail to not
reveal the encryption keys.  If I can both protect the information, and
conceal that I'm transmitting it, I'm better off.  There aren't causes
worth giving one's life for...


> I don't really understand your dual tunnel broker case.  With a normal
> tunnel broker, you get a routable address for your end of the tunnel
> (typically ::2 where the tunnel far end is ::1), and often an entire
> /64.  So just using the routable addresses would work fine.  (I have 3
> tunnels, one of which feeds a routed network of about 6 subnets carve up
> From a /48, used by my group of about 100 at work).
>

It's a pretty far out case and I don't think I did a good job explaining
it.  I would never design a network to work that way and I'm not that sad
about punishing people who (ab)use IPv6 that way.  This would also be a
case where IPv4 should be used instead of IPv6, but the way other major
IPv6 capable applications work, here's what would happen:

Let's say that my site gives out real IPv4 addresses liberally, allows me
to use tunnel brokers, but also forbids me running any DHCPv6 or router
advertisements.  Because of this I need Hurricane Electric to give me two
tunnels for two computers that I control: host1.mason.ch and host2.mason.ch.
Google chrome on host2 looks up host1.mason.ch.  It gets a few address,
let's say I have in DNS the ipv6 tunnel address for home 2001:123::1/64 and
my work address 2001:1234::1/64, and an IPv4 address of 1.2.3.4.  Google
Chrome does connect() to all different IPv6 addresses first, then IPv4
address, and then keeps the first connection it gets.  If it were on the
same IPv6 network as host1, it would not matter which address of those it
connected to, but it isn't.  It's on a separate network because I NEED two
tunnels.  Chrome (and other applications) are set up to prefer IPv6
connectivity.  In this case, I will end up communicating over two tunnels
and thousands of miles to get to the computer 2 meters of Ethernet cable
away.  It would be nice if we could prioritize IPv6 addresses on a local
computer, but we would need the whole routing table locally.  The way
Chrome gets around this is that it just take the fastest response (which
may not be the highest bandwidth response).  If I had included an fe80::
address in my DNS, Google Chrome would connect to that first because it's
the least latent link.  This is one idea behind including all possible
addresses of the host in the furl.  If Tahoe will issue connects to each
address in the list and choose the first response, or if Tahoe will issue
connects to each in order of configuration, there is the possibility of
getting the best behavior, and not just the most reliable.


>   If you bring up a host, or set of hosts, in an environment without a DHCP
>   server, and no IPv6 router, and don't run Avahi/Bonjour the only address
>   that you'll come up with is the fe80 address.  With them included, your
>   tahoe cluster can be brought up and connected to without any
> configuration,
>   without any infrastructure, it would even work with only a crossover
> cable.
>
> I really don't follow this.  You're saying that if you bring up 4
> computers with no addressing, and then somehow figure out the link-local
> address of the introducer, and put it in a furl and config files, and
> then run tahoe, that this is somehow better than manually configuring 4
> addresses (which then makes all sorts of other things easier)?  This is
> IMHO a degenerate case, and it seems odd to want to add complexity to
> tahoe to support it.  The furl is already a capability, so
> auto-discovery seems inconsistent with tahoe's security goals.
>

Link-local addresses can either be randomly generated, or deterministicly
generated, or specifically assigned.  If there is no router on your
network, and you want to run IPv6, you are only able to use FE80::
addresses.  With all the Ubuntu/Debian boxen that I have and the two OS X
boxes that my wife owns, they are always fe80:2::0 & MAC address.  I don't
know about internal representations of this in memory, but ifconfig -a is
where I see it.  I don't ever need to figure out the Link-Local addresses
of my computers because I know their mac addresses.

A further complexity is that when you look at your link-local address on
> the introducer, you'll see (assuming a:b:c:d:e:f ethernet address)
> something that looks like
>   fe80::a:b:c:d:e:f%wm0
> But if you grab that out of buffer, it will look something like
>
>   fe80::2:a:b:c:d:e:f
>
> assuming 2 is the ifindex of wm0.  I don't remember which byte is used,
> but I did figure this out in NetBSD recently.  (If my job weren't
> developing new network protocols (some of which do use link-local
> addresses), I can't imagine I would have dug into this.)  This is a
> local OS decision how to represent the interface.  BSD does this (as
> implemented by KAME), and I am 95% sure Linux does essentially the same
> thing.  So to make this work, the server has to send the fe80:: address
> without the interface ID to an introducer which is on the *same link*,
> and the introdcer can then send it to clients which are also on the same
> link.
>

As I said above, I only see the second address everywhere I've looked.  Is
this due to an old KAME version in OS X?  And as I said above I also don't
know what buffer it would be grabbed out of.

But, how does the client know which interface to use to contact the
> introducer?  So you need the client to take the actual LL address and
> then add a e.g. %wm0 scopeid when they configure their client.
>
> Strangely, on Linux, scopeid is required, on Windows and OS X, it just
works.  Ping6 fe80:: addresses to your heart's desire.


> So if you really want to use fe80:: addresses, I would say the following
> should let you do everything which will actually work, and avoid
> cluttering everyone else with them:
>
>   Users have to configure each client and server node with a LL address
>   for the introducer, and put a %intfN scopeid on it.  Of course the
>   introducer has to share a link with each node.
>
>   Nodes contact the introducer, and send their bare (no scopeid) LL
>   address.  The introducer keeps track of the incoming scopeid (it could
>   have multiple interfaces), and applies it.  When sending addresses to
>   a node, it checks that the scopeid matches the interface over which
>   that node, and if so sends the bare LL address, and if not does not
>   (because the two nodes are on different interfaces and therefore
>   different links and thus will to interoperate).
>
>
This does seem like what would be needed.  The other option would be to run
a connect() with each interface that's up as a source interface.  A little
less scientific, but maybe functional.  Is scopeid somehow global?  How
does OS X and Windows get around not using it?


> I've been running IPV6 for 15 years or so (I don't really remember when
> I brought up the 6bone connection).


Major nerd jealousy right here... hopefully I'm in your position when IPv8
comes out :-)


>  The general plan has been to have
> global addresses on systems, and to use them.  Those have varied from
> 6bone addresses to 6to4 addresses to modern addresses in 2001::.  The
> only times I deal with link-local addresses are:
>
>   looking at ndp entries
>
>   looking at ripng status
>
>   in occasional desparate times, using ssh to them to recover things.
>   (I would never try to make tahoe work this way; I would fix the
>   addresses and then have tahoe work with regular addresses.)
>
> This is a little bit like putting RFC1918 addresses in the introducer,
> where for a typical useful grid most nodes will not be able to connect
> to most RFC1918 addresses.  But people do have significant RFC1918
> privately routed networks.  They don't have routed link-local, by
> definitions.
>

The exact IPv4 analog is the 169.254.0.0/16 addresses.


> So to summarize, the only time it makes sense to use link-local
> addresses are in situations where you will be talking to only hosts that
> are on-link, and not talking to anything farther.  tahoe is almost never
> that case.  Supporting link-local in tahoe requires a lot of complexity,
> clutters the lists of addresses, and results in clients trying to make
> connections that cannot succeed.
>
> So my advice is to at least for now, limit things to global addresses.
> It's easy enough to add in LL later, but I think it will be a tremendous
> source of complexity.  And it should be optional and off by default,
> because it's an irregular thing to do.
>
> Greg



TL;DR:
I think I will have to go read up on how foolscap does the connections to
see if it does them in parallel and how often it is done.  If it is only
done once on startup and in parallel, I may leave the addresses in.  It is
functional for Windows and OS X.  If it goes through the list of addresses
in the furl every time it does any communication with a node, this may be a
bug in foolscap.

One of the reasons I'm pushing back on this is that it seems to kludge the
code to just start filtering out addresses in a big if then statement.  The
fe80 block is not an 4 bit-even block, so it's not even a simple if
addr[:4] == statement.  It also, architecturally, may not be at the Tahoe
level that this should be filtered out.  It could be better put in
foolscap.  The whole point of foolscap is to abstract away networking into
a simple RPC statement.  That's one reason why adding so little to Tahoe
got some functionality.

Would foolscap be able to hide this away for all hosts that don't just work
with link local addresses?  Is it possible to just have foolscap figure out
if Tahoe (or any foolscap service) is on any interface with that Link-Local
address?

I think I now understand why getaddrinfo returns a 4-tupple for IPv6 when
it just spit out a 2-tupple for IPv4.  The documentation says that "Note,
however, omission of *scopeid* can cause problems in manipulating scoped
IPv6 addresses."  Such a small note, such a big issue.

Once again, thanks for helping think through this.  This is far bigger an
issue than I could have imagined.  I feel so small. :-)

-Randall
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20130217/1a2ebb81/attachment-0001.html>