allow the user to specify which servers a given gateway will use for uploads

[warner]

I'd like to have a section in the client's tahoe.cfg which lets it specify the servers available for storage. In contrast to #573 (which is about runtime/per-upload specification of which servers to use, out of the set provided by the introducer), this ticket is about boot-time configuration of the available set, potentially replacing the Introducer-provided list.

My thought is that the tahoe.cfg should have a section that specifies a list of servers to use. Then another tahoe.cfg setting should have a flag which says "use the Introducer to populate this list", and the default configuration would use the Introducer. This latter section would also have a place to configure the #466-style "blesser" (a pubkey which tells the client to only accept server announcements which have been signed by the matching privkey).

This would also make it possible to configure alternative server types. The first such server type I'd like to add is an S3-based server. Regular servers would be defined with a FURL; S3 servers would be defined with a service URL and a set of authorization secrets.

The syntax I'm thinking of would look like this:

[client-server-selection]
server.X.type = tahoe-foolscap
server.X.nickname = alice
server.X.furl = pb://...
server.Y.type = tahoe-foolscap
server.Y.nickname = bob
server.Y.furl = pb://
server.Z.type = s3
server.Z.nickname = aws
server.Z.url = http://...
server.Z.creds = ...
server.Z.num_shares = 3
use_introducer = False
permute_serverids = False

The server.* lines would basically define a list of dictionaries (the "X" and "Y" strings would be discarded after tahoe.cfg is parsed).

The "use_introducer=False" line means that the client shouldn't bother talking to the Introducer. If it were True, the client would connect to the introducer and add whatever servers it knew about to the list.

The "permute_serverids=False" line means that the client shouldn't permute the serverlist on each upload. Instead, it should assign 1 (or num_shares=) shares to each server in the order they appear in this list. The total-shares "N" value ought to equal the number of servers (or rather the sum of the num_shares= values).

Having permute_serverids=False in the tahoe.cfg, rather than provided on a per-upload basis (as in #573) might prove more usable. It might be more appropriate for a fairly stable grid though: one in which new servers are not being added very frequently.

[warner]

repurposes this ticket to talk about configuring the server list in tahoe.cfg, potentially instead of using the Introducer

[zooko]

On ​the mailing list Ludovic Courtès wrote:

In my dissertation and prototype implementation, the replication
strategy is essentially a function that takes two user-defined
predicates as its arguments.  These predicates define a per-block (read
"per-share") and a per-server replication strategy.  See Section 6.3.2.1
"Replication Strategies", page 170 of the PDF at
http://tel.archives-ouvertes.fr/tel-00196822/en/ .

If I understand correctly, his dissertation includes a configuration language with which one could configure one's storage client to use certain servers in certain cases.

[zooko]

#573 has been closed as a duplicate of this. Although the original request by Shawn Willden specifically asked for dynamic -- run-time -- share placement strategies and not merely static -- start-up-time -- share placement strategies, it seems to me that most or all of the use cases that people have posted to #573 would be satisfied by the latter, and I want to drive such people (including myself) to this ticket. Please read #573!

[zooko]

USSJoin is considering using Tahoe-LAFS as the bulk storage for his gargoyle (as in Snow Crash) rig. He wants to be able to use it in caching mode for when he is offline, so that his files which are in Tahoe-LAFS are retrievable when off-line (because at least K shares of each file are stored in the local Tahoe-LAFS server which is on his person).

I think he is wrong to want this! I think that caching should be done outside of Tahoe-LAFS instead of inside of it, such as by not running a Tahoe-LAFS server on his on-person rig at all but instead having a local filesystem on there with explicitly-managed local copies of some of his files.

However, the great thing about this ticket is that it enables people to do things with Tahoe-LAFS that I don't necessarily think are a good idea. :-) In particular, USSJoin could specify that each file which is uploaded has to have at least K shares going to the one storage server which runs on his local rig.

[zooko]

Oh, see also related tickets on the subject of caching:

​http://allmydata.org/trac/tahoe-lafs/query?status=assigned&status=closed&status=new&status=reopened&order=priority&keywords=~cache

[zooko]

A particular use case for this feature has been suggested by Jake Appelbaum and also by Harold Gonzales: store files in Tahoe-LAFS in such a way that they are downloaded from publicly visible Tahoe-LAFS storage servers unless those storage servers are unavailable, in which case they are automatically downloaded from storage servers hidden inside Tor (as Tor Hidden Services).

This is a necessary improvement if we want to achieve strong censorship-resistance by integrating Tahoe-LAFS and Tor, because simply running all of your Tahoe-LAFS storage servers as Tor Hidden Services) would never do -- it would impose a significant burden on the Tor network and it would be slow and unreliable for the downloader. The hybrid approach suggested by Jake and Harold means that under normal circumstances (when that file or those storage servers are not under active attack) downloaders fetch files from the non-Hidden servers, but if that fails then downloaders automatically and transparently fall back to the fetching them from the Tor Hidden Service servers.

As of Tahoe-LAFS v1.6.0, the download side of this is working -- downloaders use the first storage servers which respond to their queries, which means they would almost never use Hidden servers unless the non-Hidden ones were missing, corrupted, or overloaded. (In Tahoe-LAFS v1.7 this download strategy will probably be further optimized and strengthened.)

But for the upload side, we need this ticket to be implemented so that we can then specify "For every file upload, put exactly 1 share on exactly K different Hidden servers, and the rest on non-Hidden servers."

[davidsarah]

Justin Stottlemyer wrote on tahoe-dev:

I could see in my case a (similar style) need. I have a concern that while running a private grid with dense storage systems containing 36-45 nodes. that I could end up with enough shares on a single system that if that system were down I could have a failure to retrieve data until that box was back up and running. A slightly different use case, but it may be able to use a similar structure.

This is a similar use case to geographic dispersal, and could probably use the same mechanism as that.

For example, the static configuration could specify a tree of nodes, and the share placement algorithm would try to distribute shares as evenly as possible between siblings at each level of the tree. In the simplest case, you would have a single-level tree and the shares would just be distributed between nodes as they are now. In a complicated case the levels of the tree could specify: continent / site / room / rack / machine / disk.

[davidsarah]

Replying to davidsarah:

In a complicated case the levels of the tree could specify: continent / site / room / rack / machine / disk.

or: storage-provider / (however that provider organizes its nodes)

[davidsarah]

The syntax I'm thinking of would look like this:

[client-server-selection]
server.X.type = tahoe-foolscap
server.X.nickname = alice
server.X.furl = pb://...
server.Y.type = tahoe-foolscap
server.Y.nickname = bob
server.Y.furl = pb://
server.Z.type = s3
server.Z.nickname = aws
server.Z.url = http://...
server.Z.creds = ...
server.Z.num_shares = 3
use_introducer = False
permute_serverids = False

The amount of boilerplate can be reduced by:

• using the nickname in place of X, Y, Z...
• having the *url property imply the protocol.

[client-server-selection]
server.alice.furl = pb://...
server.bob.furl = pb://
server.aws.s3_url = http://...
server.aws.creds = ...
server.aws.num_shares = 3
use_introducer = False
permute_serverids = False

Note that geographic dispersal is independent of using a static (introducerless) serverlist; you could do the former with or without an introducer.

[davidsarah]

More precise summary.

[zooko]

#213 was a duplicate of this one. Please read the comments in #213 for an interesting discussion.

[davidsarah]

See #999 for alternative backends. As implemented there, clients connect to servers using the existing foolscap-based storage protocol, and so the client doesn't need to know the type of backend used by a server, or the credentials it is using. I think this ticket was talking about the possibility of a client talking directly to S3 or similar services, which is not what we implemented (although it still might be interesting to allow that in future).

[leif]

I would like to be able to specify groups of servers where each group is guaranteed to receive at least K (or perhaps K+1) shares of every file, even when that means more than N shares total.

My use case is to have these groups correspond to geographic locations, so that users at each location will be able to access all of the data when their internet connection is down (and so that even if it isn't down, reads won't need to use it).

[zooko]

This was discussed at the Second Summit.

[leif]

I have a small patch at ​https://github.com/tahoe-lafs/tahoe-lafs/pull/39 that relates to this ticket. It simply moves preferred servers to the front of the server selection list, so they will be tried first.

I would rather have a servers-of-happiness-like function which would consider multiple groups of servers and require a certain number of shares placed in each group, and I would also like to be able to specify servers that the introducer doesn't know about (while also using an introducer), but this patch was a quick way to get closer to the basic functionality I wanted.

In my testing it seems to work as intended, but it is important to realize that it does not guarantee that any shares will be placed on preferred servers: If some or all of the preferred servers are down or failing to accept shares, uploads will still be considered successful as long as there are enough other servers available to satisfy servers-of-happiness.