allow the user to specify which servers a given gateway will use for uploads

[warner]

I'd like to have a section in the client's tahoe.cfg which lets it specify the servers available for storage. In contrast to #573 (which is about runtime/per-upload specification of which servers to use, out of the set provided by the introducer), this ticket is about boot-time configuration of the available set, potentially replacing the Introducer-provided list.

My thought is that the tahoe.cfg should have a section that specifies a list of servers to use. Then another tahoe.cfg setting should have a flag which says "use the Introducer to populate this list", and the default configuration would use the Introducer. This latter section would also have a place to configure the #466-style "blesser" (a pubkey which tells the client to only accept server announcements which have been signed by the matching privkey).

This would also make it possible to configure alternative server types. The first such server type I'd like to add is an S3-based server. Regular servers would be defined with a FURL; S3 servers would be defined with a service URL and a set of authorization secrets.

The syntax I'm thinking of would look like this:

[client-server-selection]
server.X.type = tahoe-foolscap
server.X.nickname = alice
server.X.furl = pb://...
server.Y.type = tahoe-foolscap
server.Y.nickname = bob
server.Y.furl = pb://
server.Z.type = s3
server.Z.nickname = aws
server.Z.url = http://...
server.Z.creds = ...
server.Z.num_shares = 3
use_introducer = False
permute_serverids = False

The server.* lines would basically define a list of dictionaries (the "X" and "Y" strings would be discarded after tahoe.cfg is parsed).

The "use_introducer=False" line means that the client shouldn't bother talking to the Introducer. If it were True, the client would connect to the introducer and add whatever servers it knew about to the list.

The "permute_serverids=False" line means that the client shouldn't permute the serverlist on each upload. Instead, it should assign 1 (or num_shares=) shares to each server in the order they appear in this list. The total-shares "N" value ought to equal the number of servers (or rather the sum of the num_shares= values).

Having permute_serverids=False in the tahoe.cfg, rather than provided on a per-upload basis (as in #573) might prove more usable. It might be more appropriate for a fairly stable grid though: one in which new servers are not being added very frequently.

[warner]

repurposes this ticket to talk about configuring the server list in tahoe.cfg, potentially instead of using the Introducer

[zooko]

On ​the mailing list Ludovic Courtès wrote:

In my dissertation and prototype implementation, the replication
strategy is essentially a function that takes two user-defined
predicates as its arguments.  These predicates define a per-block (read
"per-share") and a per-server replication strategy.  See Section 6.3.2.1
"Replication Strategies", page 170 of the PDF at
http://tel.archives-ouvertes.fr/tel-00196822/en/ .

If I understand correctly, his dissertation includes a configuration language with which one could configure one's storage client to use certain servers in certain cases.

[zooko]

#573 has been closed as a duplicate of this. Although the original request by Shawn Willden specifically asked for dynamic -- run-time -- share placement strategies and not merely static -- start-up-time -- share placement strategies, it seems to me that most or all of the use cases that people have posted to #573 would be satisfied by the latter, and I want to drive such people (including myself) to this ticket. Please read #573!

[zooko]

USSJoin is considering using Tahoe-LAFS as the bulk storage for his gargoyle (as in Snow Crash) rig. He wants to be able to use it in caching mode for when he is offline, so that his files which are in Tahoe-LAFS are retrievable when off-line (because at least K shares of each file are stored in the local Tahoe-LAFS server which is on his person).

I think he is wrong to want this! I think that caching should be done outside of Tahoe-LAFS instead of inside of it, such as by not running a Tahoe-LAFS server on his on-person rig at all but instead having a local filesystem on there with explicitly-managed local copies of some of his files.

However, the great thing about this ticket is that it enables people to do things with Tahoe-LAFS that I don't necessarily think are a good idea. :-) In particular, USSJoin could specify that each file which is uploaded has to have at least K shares going to the one storage server which runs on his local rig.

[zooko]

Oh, see also related tickets on the subject of caching:

​http://allmydata.org/trac/tahoe-lafs/query?status=assigned&status=closed&status=new&status=reopened&order=priority&keywords=~cache

[zooko]

A particular use case for this feature has been suggested by Jake Appelbaum and also by Harold Gonzales: store files in Tahoe-LAFS in such a way that they are downloaded from publicly visible Tahoe-LAFS storage servers unless those storage servers are unavailable, in which case they are automatically downloaded from storage servers hidden inside Tor (as Tor Hidden Services).

This is a necessary improvement if we want to achieve strong censorship-resistance by integrating Tahoe-LAFS and Tor, because simply running all of your Tahoe-LAFS storage servers as Tor Hidden Services) would never do -- it would impose a significant burden on the Tor network and it would be slow and unreliable for the downloader. The hybrid approach suggested by Jake and Harold means that under normal circumstances (when that file or those storage servers are not under active attack) downloaders fetch files from the non-Hidden servers, but if that fails then downloaders automatically and transparently fall back to the fetching them from the Tor Hidden Service servers.

As of Tahoe-LAFS v1.6.0, the download side of this is working -- downloaders use the first storage servers which respond to their queries, which means they would almost never use Hidden servers unless the non-Hidden ones were missing, corrupted, or overloaded. (In Tahoe-LAFS v1.7 this download strategy will probably be further optimized and strengthened.)

But for the upload side, we need this ticket to be implemented so that we can then specify "For every file upload, put exactly 1 share on exactly K different Hidden servers, and the rest on non-Hidden servers."

[davidsarah]

Justin Stottlemyer wrote on tahoe-dev:

I could see in my case a (similar style) need. I have a concern that while running a private grid with dense storage systems containing 36-45 nodes. that I could end up with enough shares on a single system that if that system were down I could have a failure to retrieve data until that box was back up and running. A slightly different use case, but it may be able to use a similar structure.

This is a similar use case to geographic dispersal, and could probably use the same mechanism as that.

For example, the static configuration could specify a tree of nodes, and the share placement algorithm would try to distribute shares as evenly as possible between siblings at each level of the tree. In the simplest case, you would have a single-level tree and the shares would just be distributed between nodes as they are now. In a complicated case the levels of the tree could specify: continent / site / room / rack / machine / disk.

[davidsarah]

Replying to davidsarah:

In a complicated case the levels of the tree could specify: continent / site / room / rack / machine / disk.

or: storage-provider / (however that provider organizes its nodes)

[davidsarah]

The syntax I'm thinking of would look like this:

[client-server-selection]
server.X.type = tahoe-foolscap
server.X.nickname = alice
server.X.furl = pb://...
server.Y.type = tahoe-foolscap
server.Y.nickname = bob
server.Y.furl = pb://
server.Z.type = s3
server.Z.nickname = aws
server.Z.url = http://...
server.Z.creds = ...
server.Z.num_shares = 3
use_introducer = False
permute_serverids = False

The amount of boilerplate can be reduced by:

• using the nickname in place of X, Y, Z...
• having the *url property imply the protocol.

[client-server-selection]
server.alice.furl = pb://...
server.bob.furl = pb://
server.aws.s3_url = http://...
server.aws.creds = ...
server.aws.num_shares = 3
use_introducer = False
permute_serverids = False

Note that geographic dispersal is independent of using a static (introducerless) serverlist; you could do the former with or without an introducer.

[davidsarah]

More precise summary.

[zooko]

#213 was a duplicate of this one. Please read the comments in #213 for an interesting discussion.

[davidsarah]

See #999 for alternative backends. As implemented there, clients connect to servers using the existing foolscap-based storage protocol, and so the client doesn't need to know the type of backend used by a server, or the credentials it is using. I think this ticket was talking about the possibility of a client talking directly to S3 or similar services, which is not what we implemented (although it still might be interesting to allow that in future).

[leif]

I would like to be able to specify groups of servers where each group is guaranteed to receive at least K (or perhaps K+1) shares of every file, even when that means more than N shares total.

My use case is to have these groups correspond to geographic locations, so that users at each location will be able to access all of the data when their internet connection is down (and so that even if it isn't down, reads won't need to use it).

[zooko]

This was discussed at the Second Summit.

[leif]

I have a small patch at ​https://github.com/tahoe-lafs/tahoe-lafs/pull/39 that relates to this ticket. It simply moves preferred servers to the front of the server selection list, so they will be tried first.

I would rather have a servers-of-happiness-like function which would consider multiple groups of servers and require a certain number of shares placed in each group, and I would also like to be able to specify servers that the introducer doesn't know about (while also using an introducer), but this patch was a quick way to get closer to the basic functionality I wanted.

In my testing it seems to work as intended, but it is important to realize that it does not guarantee that any shares will be placed on preferred servers: If some or all of the preferred servers are down or failing to accept shares, uploads will still be considered successful as long as there are enough other servers available to satisfy servers-of-happiness.

[dawuud]

Here's my introducerless branch: ​https://github.com/david415/tahoe-lafs/tree/david-introducerless It works... I've used it.

I want to clean it up a bit and write some tests. If anyone else is still interested in this feature then please take a look.

[dawuud]

truckee introducerless merged branch: ​https://github.com/david415/tahoe-lafs/tree/truckee-introducerless

Tested it on an onion grid!

[leif]

I've opened a new pull request with my preferred_peers branch rebased to today's master: ​https://github.com/tahoe-lafs/tahoe-lafs/pull/128

Btw, the introducerless branch david mentioned above was merged in to ​truckee some time ago, where there is also the multi-introducer branch discussed in #68. I hope to clean these up (write more tests) and submit a combined introducerless/multiintroducer pull request soon.

[leif]

I'm cross-posting this comment to #68 and #467.

Here is a squashed commit of the multi-introducer and introducerless patches on top of the current master: ​https://github.com/leif/tahoe-lafs/compare/master...introless-multiintro-squashed

And here is a 3-way merge combining the history of both feature branches with master in such a way that git log and git blame can still find the original commits: ​https://github.com/leif/tahoe-lafs/compare/master...introless-multiintro-with-history (creating this was a git adventure; I ended up doing the 3-way merge using -s ours and then doing another squash merge followed by git commit --amend)

I'm going to write more tests before submitting a pull request with one of these. But, if anyone wants to review or test it now I'd appreciate it!

[leif]

Here is the latest version of my preferred_peers patch:

​https://github.com/tahoe-lafs/tahoe-lafs/pull/204

[leif]

The ​preferred_peers branch landed a couple months ago.

The ​introless-multiintro branch (which is also discussed in #68) works and is up to date with master but I still have a few things I want to do with it before opening a pull request for it (tests, and moving the introless config to a JSON file).

The current thing it implements includes roughly what was described in this ticket, but (as discussed with Brian at last week's Nuts and Bolts meeting) I'm going to change the configuration interface to use a JSON file instead of hacking structured data into the key names in [client-server-selection] (as described in this ticket).

I'm tempted to add this to the 1.10.3 milestone (#68 is already there), but actually there are other things I want to do to satisfy this ticket which I don't think will happen by then (even if the introless-multiintro branch does):

• actually do what the title implies, and allow the user to specify that some servers should be used for uploads while others are read-only (a per-server configuration, plus a default for introduced servers without a local configuration).
• require that specific servers receive shares in order for an upload to be considered successful ("peers.required")

Or maybe those should be two separate tickets and this one can close if introless-multiintro lands without those features?

[daira]

leif wrote:

Or maybe those should be two separate tickets and this one can close if introless-multiintro lands without those features?

+1

[warner]

Milestone renamed

[dawuud]

I've got this dev branch here: ​https://github.com/david415/tahoe-lafs/tree/introless-multiintro_yaml_config.0

At first I made the web stuff output yaml but then I removed all the introducer-less web page config output because it's not very nice to make users copy/paste config options from a webpage. Instead I'm going to write a feature such that the yaml config is simply written to a file inside the tahoe config directory; HOWEVER we should have one yaml file per introducer. My question about this feature addition is: What should trigger the dumping of the intro'ed servers into the yaml files? Magic-Folder has a feature where the magic-folder client is not started until we have at least happiness storage servers available. We could do something like this or should we make a seperate HTTP api and CLI for this?

[dawuud]

in my dev branch i've changed the introducer client a bit to include an extra parameter in the constructor, the filename to save/load the YAML cached announcements. is this OK?

I made the introducer client save/update announcements it receives to this YAML config_file. Although it's a bit inefficient that it must load the announcement list and then append the new announcement and then write the whole thing to replace the file. But I guess this is what we get for trying to make a human editable configuration for this instead of using local database like sqlite or something.

[daira]

Reviewed at ​https://github.com/david415/tahoe-lafs/pull/6/files

[dawuud]

corrections made. please review.

[dawuud]

i made some more progress here: ​https://github.com/david415/tahoe-lafs/tree/introless-multiintro_yaml_config.0_socks_client.0

that branch has the latest tor integration dev branch merged into it so i could test with it. so far it's a work in progress because it cannot properly load a YAML storage server list file yet. I'll work on this more tomorrow.

[dawuud]

fixed! I tested it with an onion grid and it works great; i cherry-picked two of the fixing commits onto my original dev branch so that it's not mixed up with the Tor integration changes:

​https://github.com/david415/tahoe-lafs/pull/6/files

This code needs review!

I don't understand the different between a serverid and a tubid. I think we might need some code cleanup but I don't yet know all the details... but if you look... the serverid can either be derived from the key or from the tubid.

[daira]

Reviewed so far.

[daira]

I suggest asking warner the tubid vs serverid question.

[dawuud]

In order to utilize the "introducer-less" feature and make use of the previously cached announcements without connecting to the introducer the user must:

1. comment out the introducer furl from the .tahoe/private/introducers file

2. comment out the introducer furl from the .tahoe/tahoe.cfg file

3. add a "introducer.furl" to tahoe.cfg that contains the "swissnum" section of the previously used introducer furl. this results in tahoe loading announcements from a file named:

swissnum.introduced.yaml

however usage of multiple old style furls ending in the string "introducer" will not work properly with this feature set because they will cause the client's gateway to locally cache their announcements to the same file named:

introducer.introduced.yaml

Is this the desired user interface for the "introducer-less" feature?

[warner]

As mentioned on IRC, I'd like to avoid having this code synthesize a fake "serverid". In particular I want to maintain the contract that NativeStorageServer (via storage_client) gets a key_s= argument that is either the full ed25519 pubkey, or None.

So I'd suggest:

• change the stored (YAML) data format. It current stores the full announcement dictionary (plus the added 'serverid' key). Instead, I'd suggest a two-level structure, where the top dictionary has a key_s key (the value of which is exactly the first argument passed to StorageFarmBroker._got_announcement: either a full "v0-"-prefixed pubkey, or None), and an ann key (the value of which is the full announcement dictionary).
• remove the versionid changes

NativeStorageServer isn't using the key_s= argument for anything other than human-oriented descriptions yet, but I'd like to be consistent about key_s being a real pubkey (if available) so that we could use it reliably for something else in the future. Like maybe exchanging gossip with other clients about server reliability, for which having a pubkey to verify against would be handy.

[dawuud]

in the same dev branch i have made the corrections specified by warner... although i think warner meant "remove the serverid changes" in the second suggested change.

now this needs review again? am i getting close to having this changeset be acceptable?

[daira]

See also #1765.

[warner]

moving most tickets from 1.12 to 1.13 so we can release 1.12 with magic-folders

[exarkun]

Moving open issues out of closed milestones.

[meejah]

Ticket retargeted after milestone closed