source: trunk/docs/frontends/CLI.rst

Last change on this file was 64f6ccd, checked in by Itamar Turner-Trauring <itamar@…>, at 2021-05-28T13:53:28Z

Make --exclude-from behavior consistent, at the cost of a minor incompatibility.

  • Property mode set to 100644
File size: 23.1 KB
Line 
1.. -*- coding: utf-8-with-signature -*-
2
3===========================
4The Tahoe-LAFS CLI commands
5===========================
6
71.  `Overview`_
82.  `CLI Command Overview`_
9
10    1.  `Unicode Support`_
11
123.  `Node Management`_
134.  `File Store Manipulation`_
14
15    1.  `Starting Directories`_
16    2.  `Command Syntax Summary`_
17    3.  `Command Examples`_
18
195.  `Storage Grid Maintenance`_
206.  `Debugging`_
21
22
23Overview
24========
25
26Tahoe-LAFS provides a single executable named "``tahoe``", which can be used
27to create and manage client/server nodes, manipulate the file store, and
28perform several debugging/maintenance tasks. This executable is installed
29into your virtualenv when you run ``pip install tahoe-lafs``.
30
31
32CLI Command Overview
33====================
34
35The "``tahoe``" tool provides access to three categories of commands.
36
37* node management: create a client/server node, start/stop/restart it
38* file store manipulation: list files, upload, download, unlink, rename
39* debugging: unpack cap-strings, examine share files
40
41To get a list of all commands, just run "``tahoe``" with no additional
42arguments. "``tahoe --help``" might also provide something useful.
43
44Running "``tahoe --version``" will display a list of version strings, starting
45with the "allmydata" module (which contains the majority of the Tahoe-LAFS
46functionality) and including versions for a number of dependent libraries,
47like Twisted, Foolscap, cryptography, and zfec. "``tahoe --version-and-path``"
48will also show the path from which each library was imported.
49
50On Unix systems, the shell expands filename wildcards (``'*'`` and ``'?'``)
51before the program is able to read them, which may produce unexpected results
52for many ``tahoe`` comands. We recommend, if you use wildcards, to start the
53path with "``./``", for example "``tahoe cp -r ./* somewhere:``". This
54prevents the expanded filename from being interpreted as an option or as an
55alias, allowing filenames that start with a dash or contain colons to be
56handled correctly.
57
58On Windows, a single letter followed by a colon is treated as a drive
59specification rather than an alias (and is invalid unless a local path is
60allowed in that context). Wildcards cannot be used to specify multiple
61filenames to ``tahoe`` on Windows.
62
63Unicode Support
64---------------
65
66As of Tahoe-LAFS v1.7.0 (v1.8.0 on Windows), the ``tahoe`` tool supports
67non-ASCII characters in command lines and output. On Unix, the command-line
68arguments are assumed to use the character encoding specified by the
69current locale (usually given by the ``LANG`` environment variable).
70
71If a name to be output contains control characters or characters that
72cannot be represented in the encoding used on your terminal, it will be
73quoted. The quoting scheme used is similar to `POSIX shell quoting`_: in
74a "double-quoted" string, backslashes introduce escape sequences (like
75those in Python strings), but in a 'single-quoted' string all characters
76stand for themselves. This quoting is only used for output, on all
77operating systems. Your shell interprets any quoting or escapes used on
78the command line.
79
80.. _`POSIX shell quoting`: http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html
81
82
83Node Management
84===============
85
86"``tahoe create-node [NODEDIR]``" is the basic make-a-new-node
87command. It creates a new directory and populates it with files that
88will allow the "``tahoe run``" and related commands to use it later
89on. ``tahoe create-node`` creates nodes that have client functionality
90(upload/download files), web API services (controlled by the
91'[node]web.port' configuration), and storage services (unless
92``--no-storage`` is specified).
93
94NODEDIR defaults to ``~/.tahoe/`` , and newly-created nodes default to
95publishing a web server on port 3456 (limited to the loopback interface, at
96127.0.0.1, to restrict access to other programs on the same host). All of the
97other "``tahoe``" subcommands use corresponding defaults.
98
99"``tahoe create-client [NODEDIR]``" creates a node with no storage service.
100That is, it behaves like "``tahoe create-node --no-storage [NODEDIR]``".
101(This is a change from versions prior to v1.6.0.)
102
103"``tahoe create-introducer [NODEDIR]``" is used to create the Introducer node.
104This node provides introduction services and nothing else. When started, this
105node will produce a ``private/introducer.furl`` file, which should be
106published to all clients.
107
108
109Running Nodes
110-------------
111
112No matter what kind of node you created, the correct way to run it is
113to use the ``tahoe run`` command. "``tahoe run [NODEDIR]``" will start
114a previously-created node in the foreground. This command functions
115the same way on all platforms and logs to stdout. If you want to run
116the process as a daemon, it is recommended that you use your favourite
117daemonization tool.
118
119File Store Manipulation
120=======================
121
122These commands let you exmaine a Tahoe-LAFS file store, providing basic
123list/upload/download/unlink/rename/mkdir functionality. They can be used as
124primitives by other scripts. Most of these commands are fairly thin wrappers
125around web-API calls, which are described in :doc:`webapi`.
126
127By default, all file store manipulation commands look in ``~/.tahoe/`` to
128figure out which Tahoe-LAFS node they should use. When the CLI command makes
129web-API calls, it will use ``~/.tahoe/node.url`` for this purpose: a running
130Tahoe-LAFS node that provides a web-API port will write its URL into this
131file. If you want to use a node on some other host, just create ``~/.tahoe/``
132and copy that node's web-API URL into this file, and the CLI commands will
133contact that node instead of a local one.
134
135These commands also use a table of "aliases" to figure out which directory
136they ought to use a starting point. This is explained in more detail below.
137
138Starting Directories
139--------------------
140
141As described in :doc:`../architecture`, the Tahoe-LAFS distributed file store
142consists of a collection of directories and files, each of which has a
143"read-cap" or a "write-cap" (also known as a URI). Each directory is simply a
144table that maps a name to a child file or directory, and this table is turned
145into a string and stored in a mutable file. The whole set of directory and
146file "nodes" are connected together into a directed graph.
147
148To use this collection of files and directories, you need to choose a
149starting point: some specific directory that we will refer to as a
150"starting directory".  For a given starting directory, the
151"``ls [STARTING_DIR]``" command would list the contents of this directory,
152the "``ls [STARTING_DIR]/dir1``" command would look inside this directory
153for a child named "``dir1``" and list its contents,
154"``ls [STARTING_DIR]/dir1/subdir2``" would look two levels deep, etc.
155
156Note that there is no real global "root" directory, but instead each
157starting directory provides a different, possibly overlapping
158perspective on the graph of files and directories.
159
160Each Tahoe-LAFS node remembers a list of starting points, called "aliases",
161which are short Unicode strings that stand in for a directory read- or
162write- cap. They are stored (encoded as UTF-8) in the file
163``NODEDIR/private/aliases`` .  If you use the command line "``tahoe ls``"
164without any "[STARTING_DIR]" argument, then it will use the default alias,
165which is ``tahoe:``, therefore "``tahoe ls``" has the same effect as
166"``tahoe ls tahoe:``".  The same goes for the other commands that can
167reasonably use a default alias: ``get``, ``put``, ``mkdir``, ``mv``, and
168``rm``.
169
170For backwards compatibility with Tahoe-LAFS v1.0, if the ``tahoe:`` alias
171is not found in ``~/.tahoe/private/aliases``, the CLI will use the contents
172of ``~/.tahoe/private/root_dir.cap`` instead. Tahoe-LAFS v1.0 had only a
173single starting point, and stored it in this ``root_dir.cap`` file, so v1.1
174and later will use it if necessary. However, once you've set a ``tahoe:``
175alias with "``tahoe set-alias``", that will override anything in the old
176``root_dir.cap`` file.
177
178The Tahoe-LAFS CLI commands use a similar path syntax to ``scp`` and
179``rsync`` -- an optional ``ALIAS:`` prefix, followed by the pathname or
180filename. Some commands (like "``tahoe cp``") use the lack of an alias to
181mean that you want to refer to a local file, instead of something from the
182Tahoe-LAFS file store. Another way to indicate this is to start the
183pathname with "./", "~/", "~username/", or "/". On Windows, aliases
184cannot be a single character, so that it is possible to distinguish a
185path relative to an alias from a path starting with a local drive specifier.
186
187When you're dealing a single starting directory, the ``tahoe:`` alias is
188all you need. But when you want to refer to something that isn't yet
189attached to the graph rooted at that starting directory, you need to
190refer to it by its capability. The way to do that is either to use its
191capability directory as an argument on the command line, or to add an
192alias to it, with the "``tahoe add-alias``" command. Once you've added an
193alias, you can use that alias as an argument to commands.
194
195The best way to get started with Tahoe-LAFS is to create a node, start it,
196then use the following command to create a new directory and set it as your
197``tahoe:`` alias::
198
199 tahoe create-alias tahoe
200
201After that you can use "``tahoe ls tahoe:``" and
202"``tahoe cp local.txt tahoe:``", and both will refer to the directory that
203you've just created.
204
205SECURITY NOTE: For users of shared systems
206``````````````````````````````````````````
207
208Another way to achieve the same effect as the above "``tahoe create-alias``"
209command is::
210
211 tahoe add-alias tahoe `tahoe mkdir`
212
213However, command-line arguments are visible to other users (through the
214``ps`` command or ``/proc`` filesystem, or the Windows Process Explorer tool),
215so if you are using a Tahoe-LAFS node on a shared host, your login neighbors
216will be able to see (and capture) any directory caps that you set up with the
217"``tahoe add-alias``" command.
218
219The "``tahoe create-alias``" command avoids this problem by creating a new
220directory and putting the cap into your aliases file for you. Alternatively,
221you can edit the ``NODEDIR/private/aliases`` file directly, by adding a line
222like this::
223
224 fun: URI:DIR2:ovjy4yhylqlfoqg2vcze36dhde:4d4f47qko2xm5g7osgo2yyidi5m4muyo2vjjy53q4vjju2u55mfa
225
226By entering the dircap through the editor, the command-line arguments are
227bypassed, and other users will not be able to see them. Once you've added the
228alias, no other secrets are passed through the command line, so this
229vulnerability becomes less significant: they can still see your filenames and
230other arguments you type there, but not the caps that Tahoe-LAFS uses to permit
231access to your files and directories.
232
233
234Command Syntax Summary
235----------------------
236
237``tahoe add-alias ALIAS[:] DIRCAP``
238
239``tahoe create-alias ALIAS[:]``
240
241``tahoe list-aliases``
242
243``tahoe mkdir``
244
245``tahoe mkdir PATH``
246
247``tahoe ls [PATH]``
248
249``tahoe webopen [PATH]``
250
251``tahoe put [--mutable] [FROMLOCAL|-]``
252
253``tahoe put [--mutable] FROMLOCAL|- TOPATH``
254
255``tahoe put [FROMLOCAL|-] mutable-file-writecap``
256
257``tahoe get FROMPATH [TOLOCAL|-]``
258
259``tahoe cp [-r] FROMPATH TOPATH``
260
261``tahoe rm PATH``
262
263``tahoe mv FROMPATH TOPATH``
264
265``tahoe ln FROMPATH TOPATH``
266
267``tahoe backup FROMLOCAL TOPATH``
268
269In these summaries, ``PATH``, ``TOPATH`` or ``FROMPATH`` can be one of:
270
271* ``[SUBDIRS/]FILENAME`` for a path relative to the default ``tahoe:`` alias;
272* ``ALIAS:[SUBDIRS/]FILENAME`` for a path relative to another alias;
273* ``DIRCAP/[SUBDIRS/]FILENAME`` or ``DIRCAP:./[SUBDIRS/]FILENAME`` for a path
274  relative to a directory cap.
275
276See `CLI Command Overview`_ above for information on using wildcards with
277local paths, and different treatment of colons between Unix and Windows.
278
279``FROMLOCAL`` or ``TOLOCAL`` is a path in the local filesystem.
280
281
282Command Examples
283----------------
284
285``tahoe add-alias ALIAS[:] DIRCAP``
286
287 An example would be::
288
289  tahoe add-alias fun URI:DIR2:ovjy4yhylqlfoqg2vcze36dhde:4d4f47qko2xm5g7osgo2yyidi5m4muyo2vjjy53q4vjju2u55mfa
290
291 This creates an alias ``fun:`` and configures it to use the given directory
292 cap. Once this is done, "``tahoe ls fun:``" will list the contents of this
293 directory. Use "``tahoe add-alias tahoe DIRCAP``" to set the contents of the
294 default ``tahoe:`` alias.
295
296 Since Tahoe-LAFS v1.8.2, the alias name can be given with or without the
297 trailing colon.
298
299 On Windows, the alias should not be a single character, because it would be
300 confused with the drive letter of a local path.
301
302``tahoe create-alias fun``
303
304 This combines "``tahoe mkdir``" and "``tahoe add-alias``" into a single step.
305
306``tahoe list-aliases``
307
308 This displays a table of all configured aliases.
309
310``tahoe mkdir``
311
312 This creates a new empty unlinked directory, and prints its write-cap to
313 stdout. The new directory is not attached to anything else.
314
315``tahoe mkdir subdir``
316
317``tahoe mkdir /subdir``
318
319 This creates a new empty directory and attaches it below the root directory
320 of the default ``tahoe:`` alias with the name "``subdir``".
321
322``tahoe ls``
323
324``tahoe ls /``
325
326``tahoe ls tahoe:``
327
328``tahoe ls tahoe:/``
329
330 All four list the root directory of the default ``tahoe:`` alias.
331
332``tahoe ls subdir``
333
334 This lists a subdirectory of your file store.
335
336``tahoe webopen``
337
338``tahoe webopen tahoe:``
339
340``tahoe webopen tahoe:subdir/``
341
342``tahoe webopen subdir/``
343
344 This uses the python 'webbrowser' module to cause a local web browser to
345 open to the web page for the given directory. This page offers interfaces to
346 add, download, rename, and unlink files and subdirectories in that directory.
347 If no alias or path is given, this command opens the root directory of the
348 default ``tahoe:`` alias.
349
350``tahoe put file.txt``
351
352``tahoe put ./file.txt``
353
354``tahoe put /tmp/file.txt``
355
356``tahoe put ~/file.txt``
357
358 These upload the local file into the grid, and prints the new read-cap to
359 stdout. The uploaded file is not attached to any directory. All one-argument
360 forms of "``tahoe put``" perform an unlinked upload.
361
362``tahoe put -``
363
364``tahoe put``
365
366 These also perform an unlinked upload, but the data to be uploaded is taken
367 from stdin.
368
369``tahoe put file.txt uploaded.txt``
370
371``tahoe put file.txt tahoe:uploaded.txt``
372
373 These upload the local file and add it to your ``tahoe:`` root with the name
374 "``uploaded.txt``".
375
376``tahoe put file.txt subdir/foo.txt``
377
378``tahoe put - subdir/foo.txt``
379
380``tahoe put file.txt tahoe:subdir/foo.txt``
381
382``tahoe put file.txt DIRCAP/foo.txt``
383
384``tahoe put file.txt DIRCAP/subdir/foo.txt``
385
386 These upload the named file and attach them to a subdirectory of the given
387 root directory, under the name "``foo.txt``". When a directory write-cap is
388 given, you can use either ``/`` (as shown above) or ``:./`` to separate it
389 from the following path. When the source file is named "``-``", the contents
390 are taken from stdin.
391
392``tahoe put file.txt --mutable``
393
394 Create a new (SDMF) mutable file, fill it with the contents of ``file.txt``,
395 and print the new write-cap to stdout.
396
397``tahoe put file.txt MUTABLE-FILE-WRITECAP``
398
399 Replace the contents of the given mutable file with the contents of
400 ``file.txt`` and print the same write-cap to stdout.
401
402``tahoe cp file.txt tahoe:uploaded.txt``
403
404``tahoe cp file.txt tahoe:``
405
406``tahoe cp file.txt tahoe:/``
407
408``tahoe cp ./file.txt tahoe:``
409
410 These upload the local file and add it to your ``tahoe:`` root with the name
411 "``uploaded.txt``".
412
413``tahoe cp tahoe:uploaded.txt downloaded.txt``
414
415``tahoe cp tahoe:uploaded.txt ./downloaded.txt``
416
417``tahoe cp tahoe:uploaded.txt /tmp/downloaded.txt``
418
419``tahoe cp tahoe:uploaded.txt ~/downloaded.txt``
420
421 This downloads the named file from your ``tahoe:`` root, and puts the result on
422 your local filesystem.
423
424``tahoe cp tahoe:uploaded.txt fun:stuff.txt``
425
426 This copies a file from your ``tahoe:`` root to a different directory, set up
427 earlier with "``tahoe add-alias fun DIRCAP``" or "``tahoe create-alias fun``".
428
429 ``tahoe cp -r ~/my_dir/ tahoe:``
430
431 This copies the folder ``~/my_dir/`` and all its children to the grid, creating
432 the new folder ``tahoe:my_dir``. Note that the trailing slash is not required:
433 all source arguments which are directories will be copied into new
434 subdirectories of the target.
435
436 The behavior of ``tahoe cp``, like the regular UNIX ``/bin/cp``, is subtly
437 different depending upon the exact form of the arguments. In particular:
438
439* Trailing slashes indicate directories, but are not required.
440* If the target object does not already exist:
441  * and if the source is a single file, it will be copied into the target;
442  * otherwise, the target will be created as a directory.
443* If there are multiple sources, the target must be a directory.
444* If the target is a pre-existing file, the source must be a single file.
445* If the target is a directory, each source must be a named file, a named
446  directory, or an unnamed directory. It is not possible to copy an unnamed
447  file (e.g. a raw filecap) into a directory, as there is no way to know what
448  the new file should be named.
449
450
451``tahoe unlink uploaded.txt``
452
453``tahoe unlink tahoe:uploaded.txt``
454
455 This unlinks a file from your ``tahoe:`` root (that is, causes there to no
456 longer be an entry ``uploaded.txt`` in the root directory that points to it).
457 Note that this does not delete the file from the grid.
458 For backward compatibility, ``tahoe rm`` is accepted as a synonym for
459 ``tahoe unlink``.
460
461``tahoe mv uploaded.txt renamed.txt``
462
463``tahoe mv tahoe:uploaded.txt tahoe:renamed.txt``
464
465 These rename a file within your ``tahoe:`` root directory.
466
467``tahoe mv uploaded.txt fun:``
468
469``tahoe mv tahoe:uploaded.txt fun:``
470
471``tahoe mv tahoe:uploaded.txt fun:uploaded.txt``
472
473 These move a file from your ``tahoe:`` root directory to the directory
474 set up earlier with "``tahoe add-alias fun DIRCAP``" or
475 "``tahoe create-alias fun``".
476
477``tahoe backup ~ work:backups``
478
479 This command performs a versioned backup of every file and directory
480 underneath your "``~``" home directory, placing an immutable timestamped
481 snapshot in e.g. ``work:backups/Archives/2009-02-06_04:00:05Z/`` (note that
482 the timestamp is in UTC, hence the "Z" suffix), and a link to the latest
483 snapshot in work:backups/Latest/ . This command uses a small SQLite database
484 known as the "backupdb", stored in ``~/.tahoe/private/backupdb.sqlite``, to
485 remember which local files have been backed up already, and will avoid
486 uploading files that have already been backed up (except occasionally that
487 will randomly upload them again if it has been awhile since had last been
488 uploaded, just to make sure that the copy of it on the server is still good).
489 It compares timestamps and filesizes when making this comparison. It also
490 re-uses existing directories which have identical contents. This lets it
491 run faster and reduces the number of directories created.
492
493 If you reconfigure your client node to switch to a different grid, you
494 should delete the stale backupdb.sqlite file, to force "``tahoe backup``"
495 to upload all files to the new grid.
496
497 The fact that "tahoe backup" checks timestamps on your local files and
498 skips ones that don't appear to have been changed is one of the major
499 differences between "tahoe backup" and "tahoe cp -r". The other major
500 difference is that "tahoe backup" keeps links to all of the versions that
501 have been uploaded to the grid, so you can navigate among old versions
502 stored in the grid. In contrast, "tahoe cp -r" unlinks the previous
503 version from the grid directory and links the new version into place,
504 so unless you have a link to the older version stored somewhere else,
505 you'll never be able to get back to it.
506
507``tahoe backup --exclude=*~ ~ work:backups``
508
509 Same as above, but this time the backup process will ignore any
510 filename that will end with '~'. ``--exclude`` will accept any standard
511 Unix shell-style wildcards, as implemented by the
512 `Python fnmatch module <http://docs.python.org/library/fnmatch.html>`__.
513 You may give multiple ``--exclude`` options.  Please pay attention that
514 the pattern will be matched against any level of the directory tree;
515 it's still impossible to specify absolute path exclusions.
516
517``tahoe backup --exclude-from-utf-8=/path/to/filename ~ work:backups``
518
519 ``--exclude-from-utf-8`` is similar to ``--exclude``, but reads exclusion
520 patterns from a UTF-8-encoded ``/path/to/filename``, one per line.
521
522``tahoe backup --exclude-vcs ~ work:backups``
523
524 This command will ignore any file or directory name known to be used by
525 version control systems to store metadata. The excluded names are:
526
527  * CVS
528  * RCS
529  * SCCS
530  * .git
531  * .gitignore
532  * .cvsignore
533  * .svn
534  * .arch-ids
535  * {arch}
536  * =RELEASE-ID
537  * =meta-update
538  * =update
539  * .bzr
540  * .bzrignore
541  * .bzrtags
542  * .hg
543  * .hgignore
544  * _darcs
545
546Storage Grid Maintenance
547========================
548
549``tahoe manifest tahoe:``
550
551``tahoe manifest --storage-index tahoe:``
552
553``tahoe manifest --verify-cap tahoe:``
554
555``tahoe manifest --repair-cap tahoe:``
556
557``tahoe manifest --raw tahoe:``
558
559 This performs a recursive walk of the given directory, visiting every file
560 and directory that can be reached from that point. It then emits one line to
561 stdout for each object it encounters.
562
563 The default behavior is to print the access cap string (like ``URI:CHK:..``
564 or ``URI:DIR2:..``), followed by a space, followed by the full path name.
565
566 If ``--storage-index`` is added, each line will instead contain the object's
567 storage index. This (string) value is useful to determine which share files
568 (on the server) are associated with this directory tree. The ``--verify-cap``
569 and ``--repair-cap`` options are similar, but emit a verify-cap and repair-cap,
570 respectively. If ``--raw`` is provided instead, the output will be a
571 JSON-encoded dictionary that includes keys for pathnames, storage index
572 strings, and cap strings. The last line of the ``--raw`` output will be a JSON
573 encoded deep-stats dictionary.
574
575``tahoe stats tahoe:``
576
577 This performs a recursive walk of the given directory, visiting every file
578 and directory that can be reached from that point. It gathers statistics on
579 the sizes of the objects it encounters, and prints a summary to stdout.
580
581
582Debugging
583=========
584
585For a list of all debugging commands, use "``tahoe debug``". For more detailed
586help on any of these commands, use "``tahoe debug COMMAND --help``".
587
588"``tahoe debug find-shares STORAGEINDEX NODEDIRS..``" will look through one or
589more storage nodes for the share files that are providing storage for the
590given storage index.
591
592"``tahoe debug catalog-shares NODEDIRS..``" will look through one or more
593storage nodes and locate every single share they contain. It produces a report
594on stdout with one line per share, describing what kind of share it is, the
595storage index, the size of the file is used for, etc. It may be useful to
596concatenate these reports from all storage hosts and use it to look for
597anomalies.
598
599"``tahoe debug dump-share SHAREFILE``" will take the name of a single share file
600(as found by "``tahoe find-shares``") and print a summary of its contents to
601stdout. This includes a list of leases, summaries of the hash tree, and
602information from the UEB (URI Extension Block). For mutable file shares, it
603will describe which version (seqnum and root-hash) is being stored in this
604share.
605
606"``tahoe debug dump-cap CAP``" will take any Tahoe-LAFS URI and unpack it
607into separate pieces. The most useful aspect of this command is to reveal the
608storage index for any given URI. This can be used to locate the share files
609that are holding the encoded+encrypted data for this file.
610
611"``tahoe debug corrupt-share SHAREFILE``" will flip a bit in the given
612sharefile. This can be used to test the client-side verification/repair code.
613Obviously, this command should not be used during normal operation.
Note: See TracBrowser for help on using the repository browser.