1 | |
---|
2 | |
---|
3 | Managed Grid |
---|
4 | ============ |
---|
5 | |
---|
6 | This document explains the "Grid Manager" concept and the |
---|
7 | `grid-manager` command. Someone operating a grid may choose to use a |
---|
8 | Grid Manager. Operators of storage-servers and clients will then be |
---|
9 | given additional configuration in this case. |
---|
10 | |
---|
11 | |
---|
12 | Overview and Motivation |
---|
13 | ----------------------- |
---|
14 | |
---|
15 | In a grid using an Introducer, a client will use any storage-server |
---|
16 | the Introducer announces (and the Introducer will announce any |
---|
17 | storage-server that connects to it). This means that anyone with the |
---|
18 | Introducer fURL can connect storage to the grid. |
---|
19 | |
---|
20 | Sometimes, this is just what you want! |
---|
21 | |
---|
22 | For some use-cases, though, you want to have clients only use certain |
---|
23 | servers. One case might be a "managed" grid, where some entity runs |
---|
24 | the grid; clients of this grid don't want their uploads to go to |
---|
25 | "unmanaged" storage if some other client decides to provide storage. |
---|
26 | |
---|
27 | One way to limit which storage servers a client connects to is via the |
---|
28 | "server list" (:ref:`server_list`) (aka "Introducerless" |
---|
29 | mode). Clients are given static lists of storage-servers, and connect |
---|
30 | only to those. This means manually updating these lists if the storage |
---|
31 | servers change, however. |
---|
32 | |
---|
33 | Another method is for clients to use `[client] peers.preferred=` |
---|
34 | configuration option (:ref:`Client Configuration`), which suffers |
---|
35 | from a similar disadvantage. |
---|
36 | |
---|
37 | |
---|
38 | Grid Manager |
---|
39 | ------------ |
---|
40 | |
---|
41 | A "grid-manager" consists of some data defining a keypair (along with |
---|
42 | some other details) and Tahoe sub-commands to manipulate the data and |
---|
43 | produce certificates to give to storage-servers. Certificates assert |
---|
44 | the statement: "Grid Manager X suggests you use storage-server Y to |
---|
45 | upload shares to" (X and Y are public-keys). Such a certificate |
---|
46 | consists of: |
---|
47 | |
---|
48 | - the version of the format the certificate conforms to (`1`) |
---|
49 | - the public-key of a storage-server |
---|
50 | - an expiry timestamp |
---|
51 | - a signature of the above |
---|
52 | |
---|
53 | A client will always use any storage-server for downloads (expired |
---|
54 | certificate, or no certificate) because clients check the ciphertext |
---|
55 | and re-assembled plaintext against the keys in the capability; |
---|
56 | "grid-manager" certificates only control uploads. |
---|
57 | |
---|
58 | Clients make use of this functionality by configuring one or more Grid Manager public keys. |
---|
59 | This tells the client to only upload to storage-servers that have a currently-valid certificate from any of the Grid Managers their client allows. |
---|
60 | In case none are configured, the default behavior (of using any storage server) prevails. |
---|
61 | |
---|
62 | |
---|
63 | Grid Manager Data Storage |
---|
64 | ------------------------- |
---|
65 | |
---|
66 | The data defining the grid-manager is stored in an arbitrary |
---|
67 | directory, which you indicate with the ``--config`` option (in the |
---|
68 | future, we may add the ability to store the data directly in a grid, |
---|
69 | at which time you may be able to pass a directory-capability to this |
---|
70 | option). |
---|
71 | |
---|
72 | If you don't want to store the configuration on disk at all, you may |
---|
73 | use ``--config -`` (the last character is a dash) and write a valid |
---|
74 | JSON configuration to stdin. |
---|
75 | |
---|
76 | All commands require the ``--config`` option and they all behave |
---|
77 | similarly for "data from stdin" versus "data from disk". A directory |
---|
78 | (and not a file) is used on disk because in that mode, each |
---|
79 | certificate issued is also stored alongside the configuration |
---|
80 | document; in "stdin / stdout" mode, an issued certificate is only |
---|
81 | ever available on stdout. |
---|
82 | |
---|
83 | The configuration is a JSON document. It is subject to change as Grid |
---|
84 | Manager evolves. It contains a version number in the |
---|
85 | `grid_manager_config_version` key which will increment whenever the |
---|
86 | document schema changes. |
---|
87 | |
---|
88 | |
---|
89 | grid-manager create |
---|
90 | ``````````````````` |
---|
91 | |
---|
92 | Create a new grid-manager. |
---|
93 | |
---|
94 | If you specify ``--config -`` then a new grid-manager configuration is |
---|
95 | written to stdout. Otherwise, a new grid-manager is created in the |
---|
96 | directory specified by the ``--config`` option. It is an error if the |
---|
97 | directory already exists. |
---|
98 | |
---|
99 | |
---|
100 | grid-manager public-identity |
---|
101 | ```````````````````````````` |
---|
102 | |
---|
103 | Print out a grid-manager's public key. This key is derived from the |
---|
104 | private-key of the grid-manager, so a valid grid-manager config must |
---|
105 | be given via ``--config`` |
---|
106 | |
---|
107 | This public key is what is put in clients' configuration to actually |
---|
108 | validate and use grid-manager certificates. |
---|
109 | |
---|
110 | |
---|
111 | grid-manager add |
---|
112 | ```````````````` |
---|
113 | |
---|
114 | Takes two args: ``name pubkey``. The ``name`` is an arbitrary local |
---|
115 | identifier for the new storage node (also sometimes called "a petname" |
---|
116 | or "nickname"). The pubkey is the tahoe-encoded key from a ``node.pubkey`` |
---|
117 | file in the storage-server's node directory (minus any |
---|
118 | whitespace). For example, if ``~/storage0`` contains a storage-node, |
---|
119 | you might do something like this:: |
---|
120 | |
---|
121 | grid-manager --config ./gm0 add storage0 $(cat ~/storage0/node.pubkey) |
---|
122 | |
---|
123 | This adds a new storage-server to a Grid Manager's |
---|
124 | configuration. (Since it mutates the configuration, if you used |
---|
125 | ``--config -`` the new configuration will be printed to stdout). The |
---|
126 | usefulness of the ``name`` is solely for reference within this Grid |
---|
127 | Manager. |
---|
128 | |
---|
129 | |
---|
130 | grid-manager list |
---|
131 | ````````````````` |
---|
132 | |
---|
133 | Lists all storage-servers that have previously been added using |
---|
134 | ``grid-manager add``. |
---|
135 | |
---|
136 | |
---|
137 | grid-manager sign |
---|
138 | ````````````````` |
---|
139 | |
---|
140 | Takes two args: ``name expiry_days``. The ``name`` is a nickname used |
---|
141 | previously in a ``grid-manager add`` command and ``expiry_days`` is |
---|
142 | the number of days in the future when the certificate should expire. |
---|
143 | |
---|
144 | Note that this mutates the state of the grid-manager if it is on disk, |
---|
145 | by adding this certificate to our collection of issued |
---|
146 | certificates. If you used ``--config -``, the certificate isn't |
---|
147 | persisted anywhere except to stdout (so if you wish to keep it |
---|
148 | somewhere, that is up to you). |
---|
149 | |
---|
150 | This command creates a new "version 1" certificate for a |
---|
151 | storage-server (identified by its public key). The new certificate is |
---|
152 | printed to stdout. If you stored the config on disk, the new |
---|
153 | certificate will (also) be in a file named like ``alice.cert.0``. |
---|
154 | |
---|
155 | |
---|
156 | Enrolling a Storage Server: CLI |
---|
157 | ------------------------------- |
---|
158 | |
---|
159 | |
---|
160 | tahoe admin add-grid-manager-cert |
---|
161 | ````````````````````````````````` |
---|
162 | |
---|
163 | - `--filename`: the file to read the cert from |
---|
164 | - `--name`: the name of this certificate |
---|
165 | |
---|
166 | Import a "version 1" storage-certificate produced by a grid-manager A |
---|
167 | storage server may have zero or more such certificates installed; for |
---|
168 | now just one is sufficient. You will have to re-start your node after |
---|
169 | this. Subsequent announcements to the Introducer will include this |
---|
170 | certificate. |
---|
171 | |
---|
172 | .. note:: |
---|
173 | |
---|
174 | This command will simply edit the `tahoe.cfg` file and direct you |
---|
175 | to re-start. In the Future(tm), we should consider (in exarkun's |
---|
176 | words): |
---|
177 | |
---|
178 | "A python program you run as a new process" might not be the |
---|
179 | best abstraction to layer on top of the configuration |
---|
180 | persistence system, though. It's a nice abstraction for users |
---|
181 | (although most users would probably rather have a GUI) but it's |
---|
182 | not a great abstraction for automation. So at some point it |
---|
183 | may be better if there is CLI -> public API -> configuration |
---|
184 | persistence system. And maybe "public API" is even a network |
---|
185 | API for the storage server so it's equally easy to access from |
---|
186 | an agent implemented in essentially any language and maybe if |
---|
187 | the API is exposed by the storage node itself then this also |
---|
188 | gives you live-configuration-updates, avoiding the need for |
---|
189 | node restarts (not that this is the only way to accomplish |
---|
190 | this, but I think it's a good way because it avoids the need |
---|
191 | for messes like inotify and it supports the notion that the |
---|
192 | storage node process is in charge of its own configuration |
---|
193 | persistence system, not just one consumer among many ... which |
---|
194 | has some nice things going for it ... though how this interacts |
---|
195 | exactly with further node management automation might bear |
---|
196 | closer scrutiny). |
---|
197 | |
---|
198 | |
---|
199 | Enrolling a Storage Server: Config |
---|
200 | ---------------------------------- |
---|
201 | |
---|
202 | You may edit the ``[storage]`` section of the ``tahoe.cfg`` file to |
---|
203 | turn on grid-management with ``grid_management = true``. You then must |
---|
204 | also provide a ``[grid_management_certificates]`` section in the |
---|
205 | config-file which lists ``name = path/to/certificate`` pairs. |
---|
206 | |
---|
207 | These certificate files are issued by the ``grid-manager sign`` |
---|
208 | command; these should be transmitted to the storage server operator |
---|
209 | who includes them in the config for the storage server. Relative paths |
---|
210 | are based from the node directory. Example:: |
---|
211 | |
---|
212 | [storage] |
---|
213 | grid_management = true |
---|
214 | |
---|
215 | [grid_management_certificates] |
---|
216 | default = example_grid.cert |
---|
217 | |
---|
218 | This will cause us to give this certificate to any Introducers we |
---|
219 | connect to (and subsequently, the Introducer will give the certificate |
---|
220 | out to clients). |
---|
221 | |
---|
222 | |
---|
223 | Enrolling a Client: Config |
---|
224 | -------------------------- |
---|
225 | |
---|
226 | You may instruct a Tahoe client to use only storage servers from given |
---|
227 | Grid Managers. If there are no such keys, any servers are used |
---|
228 | (but see https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3979). If |
---|
229 | there are one or more keys, the client will only upload to a storage |
---|
230 | server that has a valid certificate (from any of the keys). |
---|
231 | |
---|
232 | To specify public-keys, add a ``[grid_managers]`` section to the |
---|
233 | config. This consists of ``name = value`` pairs where ``name`` is an |
---|
234 | arbitrary name and ``value`` is a public-key of a Grid |
---|
235 | Manager. Example:: |
---|
236 | |
---|
237 | [grid_managers] |
---|
238 | example_grid = pub-v0-vqimc4s5eflwajttsofisp5st566dbq36xnpp4siz57ufdavpvlq |
---|
239 | |
---|
240 | See also https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3507 which |
---|
241 | proposes a command to edit the config. |
---|
242 | |
---|
243 | |
---|
244 | Example Setup of a New Managed Grid |
---|
245 | ----------------------------------- |
---|
246 | |
---|
247 | This example creates an actual grid, but it's all just on one machine |
---|
248 | with different "node directories" and a separate tahoe process for |
---|
249 | each node. Usually of course each storage server would be on a |
---|
250 | separate computer. |
---|
251 | |
---|
252 | Note that we use the ``daemonize`` command in the following but that's |
---|
253 | only one way to handle "running a command in the background". You |
---|
254 | could instead run commands that start with ``daemonize ...`` in their |
---|
255 | own shell/terminal window or via something like ``systemd`` |
---|
256 | |
---|
257 | We'll store our Grid Manager configuration on disk, in |
---|
258 | ``./gm0``. To initialize this directory:: |
---|
259 | |
---|
260 | grid-manager --config ./gm0 create |
---|
261 | |
---|
262 | (If you already have a grid, you can :ref:`skip ahead <skip_ahead>`.) |
---|
263 | |
---|
264 | First of all, create an Introducer. Note that we actually have to run |
---|
265 | it briefly before it creates the "Introducer fURL" we want for the |
---|
266 | next steps:: |
---|
267 | |
---|
268 | tahoe create-introducer --listen=tcp --port=5555 --location=tcp:localhost:5555 ./introducer |
---|
269 | daemonize tahoe -d introducer run |
---|
270 | |
---|
271 | Next, we attach a couple of storage nodes:: |
---|
272 | |
---|
273 | tahoe create-node --introducer $(cat introducer/private/introducer.furl) --nickname storage0 --webport 6001 --location tcp:localhost:6003 --port 6003 ./storage0 |
---|
274 | tahoe create-node --introducer $(cat introducer/private/introducer.furl) --nickname storage1 --webport 6101 --location tcp:localhost:6103 --port 6103 ./storage1 |
---|
275 | daemonize tahoe -d storage0 run |
---|
276 | daemonize tahoe -d storage1 run |
---|
277 | |
---|
278 | .. _skip_ahead: |
---|
279 | |
---|
280 | We can now tell the Grid Manager about our new storage servers:: |
---|
281 | |
---|
282 | grid-manager --config ./gm0 add storage0 $(cat storage0/node.pubkey) |
---|
283 | grid-manager --config ./gm0 add storage1 $(cat storage1/node.pubkey) |
---|
284 | |
---|
285 | To produce a new certificate for each node, we do this:: |
---|
286 | |
---|
287 | grid-manager --config ./gm0 sign storage0 > ./storage0/gridmanager.cert |
---|
288 | grid-manager --config ./gm0 sign storage1 > ./storage1/gridmanager.cert |
---|
289 | |
---|
290 | Now, we want our storage servers to actually announce these |
---|
291 | certificates into the grid. We do this by adding some configuration |
---|
292 | (in ``tahoe.cfg``):: |
---|
293 | |
---|
294 | [storage] |
---|
295 | grid_management = true |
---|
296 | |
---|
297 | [grid_manager_certificates] |
---|
298 | default = gridmanager.cert |
---|
299 | |
---|
300 | Add the above bit to each node's ``tahoe.cfg`` and re-start the |
---|
301 | storage nodes. (Alternatively, use the ``tahoe add-grid-manager`` |
---|
302 | command). |
---|
303 | |
---|
304 | Now try adding a new storage server ``storage2``. This client can join |
---|
305 | the grid just fine, and announce itself to the Introducer as providing |
---|
306 | storage:: |
---|
307 | |
---|
308 | tahoe create-node --introducer $(cat introducer/private/introducer.furl) --nickname storage2 --webport 6301 --location tcp:localhost:6303 --port 6303 ./storage2 |
---|
309 | daemonize tahoe -d storage2 run |
---|
310 | |
---|
311 | At this point any client will upload to any of these three |
---|
312 | storage-servers. Make a client "alice" and try! |
---|
313 | |
---|
314 | :: |
---|
315 | |
---|
316 | tahoe create-client --introducer $(cat introducer/private/introducer.furl) --nickname alice --webport 6401 --shares-total=3 --shares-needed=2 --shares-happy=3 ./alice |
---|
317 | daemonize tahoe -d alice run |
---|
318 | tahoe -d alice put README.rst # prints out a read-cap |
---|
319 | find storage2/storage/shares # confirm storage2 has a share |
---|
320 | |
---|
321 | Now we want to make Alice only upload to the storage servers that the |
---|
322 | grid-manager has given certificates to (``storage0`` and |
---|
323 | ``storage1``). We need the grid-manager's public key to put in Alice's |
---|
324 | configuration:: |
---|
325 | |
---|
326 | grid-manager --config ./gm0 public-identity |
---|
327 | |
---|
328 | Put the key printed out above into Alice's ``tahoe.cfg`` in section |
---|
329 | ``client``:: |
---|
330 | |
---|
331 | [grid_managers] |
---|
332 | example_name = pub-v0-vqimc4s5eflwajttsofisp5st566dbq36xnpp4siz57ufdavpvlq |
---|
333 | |
---|
334 | |
---|
335 | Now, re-start the "alice" client. Since we made Alice's parameters |
---|
336 | require 3 storage servers to be reachable (``--happy=3``), all their |
---|
337 | uploads should now fail (so ``tahoe put`` will fail) because they |
---|
338 | won't use storage2 and thus can't "achieve happiness". |
---|
339 | |
---|
340 | A proposal to expose more information about Grid Manager and |
---|
341 | certificate status in the Welcome page is discussed in |
---|
342 | https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3506 |
---|