[tahoe-dev] Resurrecting Mojo Nation

[Brian Warner]

On 6/10/12 2:26 AM, James A. Donald wrote:

> Seems to me that Tahoe fails because it retreated from the goals of
> Mojo Nation - as Mojo Nation itself did.
> 
> To achieve reliability, you really want a large number of servers, for
> a large number of servers you need a large community, preferably the
> entire world as one community, but if you have large community, tahoe
> gives you the tragedy of the commons. Tahoe relies on social
> enforcement of good behavior. A community large enough to produce
> benefits from shared and distributed storage is large enough that the
> cost of informal social enforcement of good behavior is excessive -
> and bad social behavior results in unreliable storage.

Well, I'd say reliability needs enough servers to get your aggregate
uptime high enough: you need to rely upon them being up tomorrow, so you
don't have to aggressively copy data to other servers today. Having two
rock-solid servers is probably enough. Having 10 pretty-good servers is
probably enough. Having 1000 flaky servers is probably unusably bad.

The "entire world as one community" case is what we call the "one grid
to rule them all". There are a lot of engineering/scaling challenges to
it (you need log(N) DHTs, supernodes), in addition to features needed by
a mass-market product if you want that kind of scale (NAT handling,
bandwidth limiting, automatic update, nice UI, easy installation, etc,
etc). But the real challenge is the server-selection. Like you said,
once the grid gets large, social mechanisms no longer suffice, and a
strong economic model is the only obvious alternative.

We've tried to limit the scope of Tahoe to grids on which social
mechanisms are still effective, which is why we've focussed more on
"friendnets" and sub-Dunbar-number (<=200ish) scaling mechanisms. I
think we have some good ideas on the other engineering/scaling tools
needed for the one-grid-to-rule-them-all case, but I really don't think
we have an answer for the uptime/reliability part.

> Also, tahoe abandons EGTP.  We need a replacement for TCP that provides
> end to end encryption, protocol negotiation, and NAT tunneling.  TCP is
> an unworkable protocol in a large, untrusting, and untrustworthy
> society.  EGTP was such a replacement, or was intended to be such a
> replacement.

Hm. There are a bunch of problems with TCP (NAT being the biggest
hassle), but I'm not sure I'd blame it for encryption and protocol
negotiation problems.. any security-conscious app will build those
pieces on top of it, resulting in something roughly the same size as
EGTP. That said, the flexibility of routing and message-passing *is*
prompting us to move away from connection-oriented protocols (Foolscap)
towards more discrete-encrypted-message forwarding schemes, so I think
within a year or two we'll have something that looks more like EGTP (but
with faster modern crypto). It will still be delivered over pairwise TCP
connections, though (which I think EGTP was too).

> I would say that the main result of the Tahoe experiment is that you
> really have to do what Mojo Nation attempted to do, and that the
> subset has problems.

Heh, I'd say the result was the opposite: Mojo Nation attempted to do a
lot and failed, Tahoe attempted to much less and succeeded. If Mojo
Nation had been implementable and managed to work as intended, it could
probably handle the large-grid case better than Tahoe ever will. But in
attempting to do that, they built a hill so large that they couldn't
climb it.


cheers,
 -Brian