#467 new enhancement

allow the user to specify which servers a given gateway will use for uploads

Reported by: warner Owned by: leif
Priority: major Milestone: 1.15.0
Component: code-peerselection Version: 1.1.0
Keywords: availability preservation cache anti-censorship placement backend rollback add-only Cc: ussjoin@…, leif@…, matt@…
Launchpad Bug:

Description (last modified by zooko)

I'd like to have a section in the client's tahoe.cfg which lets it specify the servers available for storage. In contrast to #573 (which is about runtime/per-upload specification of which servers to use, out of the set provided by the introducer), this ticket is about boot-time configuration of the available set, potentially replacing the Introducer-provided list.

My thought is that the tahoe.cfg should have a section that specifies a list of servers to use. Then another tahoe.cfg setting should have a flag which says "use the Introducer to populate this list", and the default configuration would use the Introducer. This latter section would also have a place to configure the #466-style "blesser" (a pubkey which tells the client to only accept server announcements which have been signed by the matching privkey).

This would also make it possible to configure alternative server types. The first such server type I'd like to add is an S3-based server. Regular servers would be defined with a FURL; S3 servers would be defined with a service URL and a set of authorization secrets.

The syntax I'm thinking of would look like this:

server.X.type = tahoe-foolscap
server.X.nickname = alice
server.X.furl = pb://...
server.Y.type = tahoe-foolscap
server.Y.nickname = bob
server.Y.furl = pb://
server.Z.type = s3
server.Z.nickname = aws
server.Z.url = http://...
server.Z.creds = ...
server.Z.num_shares = 3
use_introducer = False
permute_serverids = False

The server.* lines would basically define a list of dictionaries (the "X" and "Y" strings would be discarded after tahoe.cfg is parsed).

The "use_introducer=False" line means that the client shouldn't bother talking to the Introducer. If it were True, the client would connect to the introducer and add whatever servers it knew about to the list.

The "permute_serverids=False" line means that the client shouldn't permute the serverlist on each upload. Instead, it should assign 1 (or num_shares=) shares to each server in the order they appear in this list. The total-shares "N" value ought to equal the number of servers (or rather the sum of the num_shares= values).

Having permute_serverids=False in the tahoe.cfg, rather than provided on a per-upload basis (as in #573) might prove more usable. It might be more appropriate for a fairly stable grid though: one in which new servers are not being added very frequently.

Change History (50)

comment:1 Changed at 2009-04-06T07:18:30Z by warner

  • Description modified (diff)
  • Summary changed from change peer-selection to prepare for rights-amplification step, alternative backends to change peer-selection to allow introducerless explicit serverlist, alternative backends

repurposes this ticket to talk about configuring the server list in tahoe.cfg, potentially instead of using the Introducer

comment:2 Changed at 2009-08-11T13:23:05Z by zooko

On the mailing list Ludovic Courtès wrote:

In my dissertation and prototype implementation, the replication
strategy is essentially a function that takes two user-defined
predicates as its arguments.  These predicates define a per-block (read
"per-share") and a per-server replication strategy.  See Section
"Replication Strategies", page 170 of the PDF at
http://tel.archives-ouvertes.fr/tel-00196822/en/ .

If I understand correctly, his dissertation includes a configuration language with which one could configure one's storage client to use certain servers in certain cases.

comment:3 Changed at 2010-01-07T03:59:22Z by davidsarah

  • Keywords availability preservation added

comment:4 Changed at 2010-02-20T04:03:39Z by zooko

#573 has been closed as a duplicate of this. Although the original request by Shawn Willden specifically asked for dynamic -- run-time -- share placement strategies and not merely static -- start-up-time -- share placement strategies, it seems to me that most or all of the use cases that people have posted to #573 would be satisfied by the latter, and I want to drive such people (including myself) to this ticket. Please read #573!

comment:5 Changed at 2010-02-20T04:26:05Z by zooko

USSJoin is considering using Tahoe-LAFS as the bulk storage for his gargoyle (as in Snow Crash) rig. He wants to be able to use it in caching mode for when he is offline, so that his files which are in Tahoe-LAFS are retrievable when off-line (because at least K shares of each file are stored in the local Tahoe-LAFS server which is on his person).

I think he is wrong to want this! I think that caching should be done outside of Tahoe-LAFS instead of inside of it, such as by not running a Tahoe-LAFS server on his on-person rig at all but instead having a local filesystem on there with explicitly-managed local copies of some of his files.

However, the great thing about this ticket is that it enables people to do things with Tahoe-LAFS that I don't necessarily think are a good idea. :-) In particular, USSJoin could specify that each file which is uploaded has to have at least K shares going to the one storage server which runs on his local rig.

comment:6 Changed at 2010-02-20T04:30:37Z by davidsarah

  • Keywords cache added

comment:8 Changed at 2010-02-20T05:07:37Z by zooko

A particular use case for this feature has been suggested by Jake Appelbaum and also by Harold Gonzales: store files in Tahoe-LAFS in such a way that they are downloaded from publicly visible Tahoe-LAFS storage servers unless those storage servers are unavailable, in which case they are automatically downloaded from storage servers hidden inside Tor (as Tor Hidden Services).

This is a necessary improvement if we want to achieve strong censorship-resistance by integrating Tahoe-LAFS and Tor, because simply running all of your Tahoe-LAFS storage servers as Tor Hidden Services) would never do -- it would impose a significant burden on the Tor network and it would be slow and unreliable for the downloader. The hybrid approach suggested by Jake and Harold means that under normal circumstances (when that file or those storage servers are not under active attack) downloaders fetch files from the non-Hidden servers, but if that fails then downloaders automatically and transparently fall back to the fetching them from the Tor Hidden Service servers.

As of Tahoe-LAFS v1.6.0, the download side of this is working -- downloaders use the first storage servers which respond to their queries, which means they would almost never use Hidden servers unless the non-Hidden ones were missing, corrupted, or overloaded. (In Tahoe-LAFS v1.7 this download strategy will probably be further optimized and strengthened.)

But for the upload side, we need this ticket to be implemented so that we can then specify "For every file upload, put exactly 1 share on exactly K different Hidden servers, and the rest on non-Hidden servers."

comment:9 follow-up: Changed at 2010-02-20T07:47:02Z by davidsarah

  • Milestone changed from undecided to eventually

Justin Stottlemyer wrote on tahoe-dev:

I could see in my case a (similar style) need. I have a concern that while running a private grid with dense storage systems containing 36-45 nodes. that I could end up with enough shares on a single system that if that system were down I could have a failure to retrieve data until that box was back up and running. A slightly different use case, but it may be able to use a similar structure.

This is a similar use case to geographic dispersal, and could probably use the same mechanism as that.

For example, the static configuration could specify a tree of nodes, and the share placement algorithm would try to distribute shares as evenly as possible between siblings at each level of the tree. In the simplest case, you would have a single-level tree and the shares would just be distributed between nodes as they are now. In a complicated case the levels of the tree could specify: continent / site / room / rack / machine / disk.

comment:10 in reply to: ↑ 9 Changed at 2010-02-20T07:50:35Z by davidsarah

Replying to davidsarah:

In a complicated case the levels of the tree could specify: continent / site / room / rack / machine / disk.

or: storage-provider / (however that provider organizes its nodes)

comment:11 Changed at 2010-02-20T20:19:42Z by davidsarah

The syntax I'm thinking of would look like this:

server.X.type = tahoe-foolscap
server.X.nickname = alice
server.X.furl = pb://...
server.Y.type = tahoe-foolscap
server.Y.nickname = bob
server.Y.furl = pb://
server.Z.type = s3
server.Z.nickname = aws
server.Z.url = http://...
server.Z.creds = ...
server.Z.num_shares = 3
use_introducer = False
permute_serverids = False

The amount of boilerplate can be reduced by:

  • using the nickname in place of X, Y, Z...
  • having the *url property imply the protocol.
server.alice.furl = pb://...
server.bob.furl = pb://
server.aws.s3_url = http://...
server.aws.creds = ...
server.aws.num_shares = 3
use_introducer = False
permute_serverids = False

Note that geographic dispersal is independent of using a static (introducerless) serverlist; you could do the former with or without an introducer.

comment:12 Changed at 2010-02-23T00:11:39Z by USSJoin

  • Cc ussjoin@… added

comment:13 Changed at 2010-02-27T06:31:55Z by zooko

  • Milestone changed from eventually to 1.7.0

comment:14 Changed at 2010-03-20T05:05:43Z by zooko

  • Summary changed from change peer-selection to allow introducerless explicit serverlist, alternative backends to allow the user to specify which servers are used for uploads

comment:15 Changed at 2010-03-20T22:44:10Z by davidsarah

  • Summary changed from allow the user to specify which servers are used for uploads to allow the user to specify which servers a given gateway will use for uploads

More precise summary.

comment:16 Changed at 2010-06-16T04:02:05Z by davidsarah

  • Milestone changed from 1.7.0 to soon

comment:17 Changed at 2010-12-16T01:14:59Z by davidsarah

  • Keywords anti-censorship added

comment:18 Changed at 2010-12-29T14:46:29Z by zooko

#213 was a duplicate of this one. Please read the comments in #213 for an interesting discussion.

comment:19 Changed at 2011-08-27T01:41:31Z by davidsarah

  • Keywords placement backend added

comment:20 Changed at 2011-10-06T19:15:28Z by davidsarah

See #999 for alternative backends. As implemented there, clients connect to servers using the existing foolscap-based storage protocol, and so the client doesn't need to know the type of backend used by a server, or the credentials it is using. I think this ticket was talking about the possibility of a client talking directly to S3 or similar services, which is not what we implemented (although it still might be interesting to allow that in future).

comment:21 Changed at 2011-11-11T09:01:50Z by leif

I would like to be able to specify groups of servers where each group is guaranteed to receive at least K (or perhaps K+1) shares of every file, even when that means more than N shares total.

My use case is to have these groups correspond to geographic locations, so that users at each location will be able to access all of the data when their internet connection is down (and so that even if it isn't down, reads won't need to use it).

comment:22 Changed at 2012-01-17T06:40:00Z by zooko

  • Description modified (diff)

comment:23 Changed at 2012-04-18T21:51:52Z by zooko

This was discussed at the Second Summit.

comment:24 Changed at 2013-04-18T01:09:25Z by leif

I have a small patch at https://github.com/tahoe-lafs/tahoe-lafs/pull/39 that relates to this ticket. It simply moves preferred servers to the front of the server selection list, so they will be tried first.

I would rather have a servers-of-happiness-like function which would consider multiple groups of servers and require a certain number of shares placed in each group, and I would also like to be able to specify servers that the introducer doesn't know about (while also using an introducer), but this patch was a quick way to get closer to the basic functionality I wanted.

In my testing it seems to work as intended, but it is important to realize that it does not guarantee that any shares will be placed on preferred servers: If some or all of the preferred servers are down or failing to accept shares, uploads will still be considered successful as long as there are enough other servers available to satisfy servers-of-happiness.

comment:25 Changed at 2013-04-18T01:16:23Z by leif

  • Cc leif@… added

comment:26 Changed at 2013-11-13T05:13:03Z by zooko

  • Description modified (diff)
  • Keywords rollback add-only added

comment:27 Changed at 2014-01-15T21:54:29Z by dawuud

Here's my introducerless branch: https://github.com/david415/tahoe-lafs/tree/david-introducerless It works... I've used it.

I want to clean it up a bit and write some tests. If anyone else is still interested in this feature then please take a look.

comment:28 Changed at 2014-01-16T14:34:47Z by dawuud

truckee introducerless merged branch: https://github.com/david415/tahoe-lafs/tree/truckee-introducerless

Tested it on an onion grid!

comment:29 Changed at 2014-11-22T20:53:03Z by leif

I've opened a new pull request with my preferred_peers branch rebased to today's master: https://github.com/tahoe-lafs/tahoe-lafs/pull/128

Btw, the introducerless branch david mentioned above was merged in to truckee some time ago, where there is also the multi-introducer branch discussed in #68. I hope to clean these up (write more tests) and submit a combined introducerless/multiintroducer pull request soon.

comment:30 Changed at 2014-11-24T01:50:33Z by leif

I'm cross-posting this comment to #68 and #467.

Here is a squashed commit of the multi-introducer and introducerless patches on top of the current master: https://github.com/leif/tahoe-lafs/compare/master...introless-multiintro-squashed

And here is a 3-way merge combining the history of both feature branches with master in such a way that git log and git blame can still find the original commits: https://github.com/leif/tahoe-lafs/compare/master...introless-multiintro-with-history (creating this was a git adventure; I ended up doing the 3-way merge using -s ours and then doing another squash merge followed by git commit --amend)

I'm going to write more tests before submitting a pull request with one of these. But, if anyone wants to review or test it now I'd appreciate it!

comment:31 Changed at 2014-12-30T07:16:57Z by mhazinsk

  • Cc matt@… added

comment:32 Changed at 2015-12-01T20:01:48Z by leif

Here is the latest version of my preferred_peers patch:


comment:33 Changed at 2016-01-31T21:00:07Z by leif

The preferred_peers branch landed a couple months ago.

The introless-multiintro branch (which is also discussed in #68) works and is up to date with master but I still have a few things I want to do with it before opening a pull request for it (tests, and moving the introless config to a JSON file).

The current thing it implements includes roughly what was described in this ticket, but (as discussed with Brian at last week's Nuts and Bolts meeting) I'm going to change the configuration interface to use a JSON file instead of hacking structured data into the key names in [client-server-selection] (as described in this ticket).

I'm tempted to add this to the 1.10.3 milestone (#68 is already there), but actually there are other things I want to do to satisfy this ticket which I don't think will happen by then (even if the introless-multiintro branch does):

  • actually do what the title implies, and allow the user to specify that some servers should be used for uploads while others are read-only (a per-server configuration, plus a default for introduced servers without a local configuration).
  • require that specific servers receive shares in order for an upload to be considered successful ("peers.required")

Or maybe those should be two separate tickets and this one can close if introless-multiintro lands without those features?

comment:34 Changed at 2016-02-02T17:35:36Z by daira

  • Milestone changed from soon to 1.11.0
  • Owner set to leif

comment:35 Changed at 2016-03-21T15:34:11Z by daira

leif wrote:

Or maybe those should be two separate tickets and this one can close if introless-multiintro lands without those features?


comment:36 Changed at 2016-03-22T05:02:52Z by warner

  • Milestone changed from 1.11.0 to 1.12.0

Milestone renamed

comment:37 Changed at 2016-03-23T17:45:41Z by dawuud

I've got this dev branch here: https://github.com/david415/tahoe-lafs/tree/introless-multiintro_yaml_config.0

At first I made the web stuff output yaml but then I removed all the introducer-less web page config output because it's not very nice to make users copy/paste config options from a webpage. Instead I'm going to write a feature such that the yaml config is simply written to a file inside the tahoe config directory; HOWEVER we should have one yaml file per introducer. My question about this feature addition is: What should trigger the dumping of the intro'ed servers into the yaml files? Magic-Folder has a feature where the magic-folder client is not started until we have at least happiness storage servers available. We could do something like this or should we make a seperate HTTP api and CLI for this?

comment:38 Changed at 2016-03-24T11:07:05Z by dawuud

in my dev branch i've changed the introducer client a bit to include an extra parameter in the constructor, the filename to save/load the YAML cached announcements. is this OK?

I made the introducer client save/update announcements it receives to this YAML config_file. Although it's a bit inefficient that it must load the announcement list and then append the new announcement and then write the whole thing to replace the file. But I guess this is what we get for trying to make a human editable configuration for this instead of using local database like sqlite or something.

comment:40 Changed at 2016-03-25T00:37:57Z by dawuud

corrections made. please review.

comment:41 Changed at 2016-03-25T02:14:47Z by dawuud

i made some more progress here: https://github.com/david415/tahoe-lafs/tree/introless-multiintro_yaml_config.0_socks_client.0

that branch has the latest tor integration dev branch merged into it so i could test with it. so far it's a work in progress because it cannot properly load a YAML storage server list file yet. I'll work on this more tomorrow.

comment:42 Changed at 2016-03-25T08:44:55Z by dawuud

fixed! I tested it with an onion grid and it works great; i cherry-picked two of the fixing commits onto my original dev branch so that it's not mixed up with the Tor integration changes:


This code needs review!

I don't understand the different between a serverid and a tubid. I think we might need some code cleanup but I don't yet know all the details... but if you look... the serverid can either be derived from the key or from the tubid.

Last edited at 2016-03-25T08:50:41Z by dawuud (previous) (diff)

comment:43 Changed at 2016-03-25T11:18:39Z by daira

Reviewed so far.

comment:44 Changed at 2016-03-25T11:19:48Z by daira

I suggest asking warner the tubid vs serverid question.

comment:45 Changed at 2016-03-26T12:27:23Z by dawuud

In order to utilize the "introducer-less" feature and make use of the previously cached announcements without connecting to the introducer the user must:

  1. comment out the introducer furl from the .tahoe/private/introducers file
  1. comment out the introducer furl from the .tahoe/tahoe.cfg file
  1. add a "introducer.furl" to tahoe.cfg that contains the "swissnum" section of the previously used introducer furl. this results in tahoe loading announcements from a file named:


however usage of multiple old style furls ending in the string "introducer" will not work properly with this feature set because they will cause the client's gateway to locally cache their announcements to the same file named:


Is this the desired user interface for the "introducer-less" feature?

comment:46 Changed at 2016-03-26T20:38:59Z by warner

As mentioned on IRC, I'd like to avoid having this code synthesize a fake "serverid". In particular I want to maintain the contract that NativeStorageServer (via storage_client) gets a key_s= argument that is either the full ed25519 pubkey, or None.

So I'd suggest:

  • change the stored (YAML) data format. It current stores the full announcement dictionary (plus the added 'serverid' key). Instead, I'd suggest a two-level structure, where the top dictionary has a key_s key (the value of which is exactly the first argument passed to StorageFarmBroker._got_announcement: either a full "v0-"-prefixed pubkey, or None), and an ann key (the value of which is the full announcement dictionary).
  • remove the versionid changes

NativeStorageServer isn't using the key_s= argument for anything other than human-oriented descriptions yet, but I'd like to be consistent about key_s being a real pubkey (if available) so that we could use it reliably for something else in the future. Like maybe exchanging gossip with other clients about server reliability, for which having a pubkey to verify against would be handy.

comment:47 Changed at 2016-03-27T08:35:56Z by dawuud

in the same dev branch i have made the corrections specified by warner... although i think warner meant "remove the serverid changes" in the second suggested change.

now this needs review again? am i getting close to having this changeset be acceptable?

comment:48 Changed at 2016-03-31T14:34:47Z by daira

See also #1765.

comment:49 Changed at 2016-06-28T18:20:37Z by warner

  • Milestone changed from 1.12.0 to 1.13.0

moving most tickets from 1.12 to 1.13 so we can release 1.12 with magic-folders

comment:50 Changed at 2020-06-30T14:45:13Z by exarkun

  • Milestone changed from 1.13.0 to 1.15.0

Moving open issues out of closed milestones.

Note: See TracTickets for help on using tickets.