[tahoe-lafs-trac-stream] [tahoe-lafs] #467: allow the user to specify which servers a given gateway will use for uploads

[tahoe-lafs]

#467: allow the user to specify which servers a given gateway will use for
uploads
-------------------------+-------------------------------------------------
     Reporter:  warner   |      Owner:
         Type:           |     Status:  new
  enhancement            |  Milestone:  soon
     Priority:  major    |    Version:  1.1.0
    Component:  code-    |   Keywords:  availability preservation cache
  peerselection          |  anti-censorship placement backend
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Old description:

> I'd like to have a section in the client's tahoe.cfg which lets it
> specify the servers available for storage. In contrast to #573 (which is
> about runtime/per-upload specification of which servers to use, out of
> the set provided by the introducer), this ticket is about boot-time
> configuration of the available set, potentially replacing the
> Introducer-provided list.
>
> My thought is that the tahoe.cfg should have a section that specifies a
> list of servers to use. Then another tahoe.cfg setting should have a
> flag which says "use the Introducer to populate this list", and the
> default configuration would use the Introducer. This latter section
> would also have a place to configure the #466-style "blesser" (a pubkey
> which tells the client to only accept server announcements which have
> been signed by the matching privkey).
>
> This would also make it possible to configure alternative server types.
> The first such server type I'd like to add is an S3-based server.
> Regular servers would be defined with a FURL; S3 servers would be
> defined with a service URL and a set of authorization secrets.
>
> The syntax I'm thinking of would look like this:
>
> {{{
> [client-server-selection]
> server.X.type = tahoe-foolscap
> server.X.nickname = alice
> server.X.furl = pb://...
> server.Y.type = tahoe-foolscap
> server.Y.nickname = bob
> server.Y.furl = pb://
> server.Z.type = s3
> server.Z.nickname = aws
> server.Z.url = http://...
> server.Z.creds = ...
> server.Z.num_shares = 3
> use_introducer = False
> permute_serverids = False
> }}}
>
> The {{{server.*}}} lines would basically define a list of dictionaries
> (the "X" and "Y" strings would be discarded after tahoe.cfg is parsed).
>
> The "use_introducer=False" line means that the client shouldn't bother
> talking to the Introducer. If it were True, the client would connect to
> the introducer and add whatever servers it knew about to the list.
>
> The "permute_serverids=False" line means that the client shouldn't
> permute the serverlist on each upload. Instead, it should assign 1 (or
> num_shares=) shares to each server in the order they appear in this
> list. The total-shares "N" value ought to equal the number of servers
> (or rather the sum of the num_shares= values).
>
> Having permute_serverids=False in the tahoe.cfg, rather than provided on
> a per-upload basis (as in #573) might prove more usable. It might be
> more appropriate for a fairly stable grid though: one in which new
> servers are not being added very frequently.

New description:

 I'd like to have a section in the client's tahoe.cfg which lets it
 specify the servers available for storage. In contrast to #573 (which is
 about runtime/per-upload specification of which servers to use, out of
 the set provided by the introducer), this ticket is about boot-time
 configuration of the available set, potentially replacing the
 Introducer-provided list.

 My thought is that the tahoe.cfg should have a section that specifies a
 list of servers to use. Then another tahoe.cfg setting should have a
 flag which says "use the Introducer to populate this list", and the
 default configuration would use the Introducer. This latter section
 would also have a place to configure the #466-style "blesser" (a pubkey
 which tells the client to only accept server announcements which have
 been signed by the matching privkey).

 This would also make it possible to configure alternative server types.
 The first such server type I'd like to add is an S3-based server.
 Regular servers would be defined with a FURL; S3 servers would be
 defined with a service URL and a set of authorization secrets.

 The syntax I'm thinking of would look like this:

 {{{
 [client-server-selection]
 server.X.type = tahoe-foolscap
 server.X.nickname = alice
 server.X.furl = pb://...
 server.Y.type = tahoe-foolscap
 server.Y.nickname = bob
 server.Y.furl = pb://
 server.Z.type = s3
 server.Z.nickname = aws
 server.Z.url = http://...
 server.Z.creds = ...
 server.Z.num_shares = 3
 use_introducer = False
 permute_serverids = False
 }}}

 The {{{server.*}}} lines would basically define a list of dictionaries
 (the "X" and "Y" strings would be discarded after tahoe.cfg is parsed).

 The "use_introducer=False" line means that the client shouldn't bother
 talking to the Introducer. If it were True, the client would connect to
 the introducer and add whatever servers it knew about to the list.

 The "permute_serverids=False" line means that the client shouldn't
 permute the serverlist on each upload. Instead, it should assign 1 (or
 num_shares=) shares to each server in the order they appear in this
 list. The total-shares "N" value ought to equal the number of servers
 (or rather the sum of the num_shares= values).

 Having permute_serverids=False in the tahoe.cfg, rather than provided on
 a per-upload basis (as in #573) might prove more usable. It might be
 more appropriate for a fairly stable grid though: one in which new
 servers are not being added very frequently.

--

Comment (by leif):

 I have a small patch at https://github.com/tahoe-lafs/tahoe-lafs/pull/39
 that relates to this ticket. It simply moves preferred servers to the
 front of the server selection list, so they will be tried first.

 I would rather have a servers-of-happiness-like function which would
 consider multiple groups of servers and require a certain number of shares
 placed in each group, and I would also like to be able to specify servers
 that the introducer doesn't know about (while also using an introducer),
 but this patch was a quick way to get closer to the basic functionality I
 wanted.

 In my testing it seems to work as intended, but it is important to realize
 that it does not guarantee that any shares will be placed on preferred
 servers: If some or all of the preferred servers are down or failing to
 accept shares, uploads will still be considered successful as long as
 there are enough other servers available to satisfy servers-of-happiness.

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/467#comment:24>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage