users are confused by "tahoe rm"

[zooko]

I found the following conversation log between a new user (PovAddict) and an experienced user (soul9). They are confused about what "tahoe rm" means. Possible changes that could improve this include renaming it to "tahoe unlink", improving the error messages (see below), and improving the docs.

<PovAddict> I just installed a tahoe client and attached it to the "volunteer grid"							[19:01]
...
<PovAddict> also, too many Python exceptions to take tahoe seriously...	[19:17]
<soul9> PovAddict: what python exceptions.				[19:19]
<soul9> ?
<soul9> python2.6?
<PovAddict> 2.5.2
<soul9> hm
<soul9> i didn't have python exceptions, except deprecationwarnings in python
	2.6
<PovAddict> http://pastebin.com/df3bfddf				[19:20]

      nicolas@ubuntu /tmp/tahoe-c$ ~/src/allmydata-tahoe-1.4.1/bin/tahoe ls
      Traceback (most recent call last):
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/support/bin/tahoe", line 8, in <module>
          load_entry_point('allmydata-tahoe==1.4.1', 'console_scripts', 'tahoe')()
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/runner.py", line 91, in run
          rc = runner(sys.argv[1:])
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/runner.py", line 78, in runner
          rc = cli.dispatch[command](so)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/cli.py", line 380, in list
          rc = tahoe_ls.list(options)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/tahoe_ls.py", line 18, in list
          rootcap, path = get_alias(aliases, where, DEFAULT_ALIAS)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/common.py", line 148, in get_alias
          return aliases[default], path
      KeyError: 'tahoe'
      nicolas@ubuntu /tmp/tahoe-c$ ~/src/allmydata-tahoe-1.4.1/bin/tahoe ls URI:CHK:2nxmn6eew3y5e5y7ejzwfr2jiq:6mi2hcbpuvkfcttuezi4x3fxwrnmwidql2rtwwtczpccxgz3ez2q:3:10:310946
      Traceback (most recent call last):
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/support/bin/tahoe", line 8, in <module>
          load_entry_point('allmydata-tahoe==1.4.1', 'console_scripts', 'tahoe')()
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/runner.py", line 91, in run
          rc = runner(sys.argv[1:])
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/runner.py", line 78, in runner
          rc = cli.dispatch[command](so)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/cli.py", line 380, in list
          rc = tahoe_ls.list(options)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/tahoe_ls.py", line 67, in list
          childtype = child[0]
      KeyError: 0
      nicolas@ubuntu /tmp/tahoe-c$ ~/src/allmydata-tahoe-1.4.1/bin/tahoe rm URI:CHK:2nxmn6eew3y5e5y7ejzwfr2jiq:6mi2hcbpuvkfcttuezi4x3fxwrnmwidql2rtwwtczpccxgz3ez2q:3:10:310946
      Traceback (most recent call last):
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/support/bin/tahoe", line 8, in <module>
          load_entry_point('allmydata-tahoe==1.4.1', 'console_scripts', 'tahoe')()
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/runner.py", line 91, in run
          rc = runner(sys.argv[1:])
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/runner.py", line 78, in runner
          rc = cli.dispatch[command](so)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/cli.py", line 409, in rm
          rc = tahoe_rm.rm(options)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/tahoe_rm.py", line 25, in rm
          assert path
      AssertionError

<PovAddict> I uploaded a file to the VolunteerGrid from the web interface
<PovAddict> I can download it back from there, or using "tahoe get"
<soul9> yes, well
<PovAddict> and that's it; is 'rm' supposed to work?
<soul9> i think that has to do with a not defined default rootcap	[19:21]
<soul9> yes, it is, of course
<PovAddict> no matter what the problem really is, the command line tool should
	    give a proper error message, instead of dumping a stacktrace...
<soul9> er								[19:22]
<PovAddict> "KeyError: 0" might only mean something to people who know the
	    code and know the causes of this situation
<PovAddict> definitely means nothing to me
<soul9> it's not dumping it's core at all
<PovAddict> it printed a stacktrace on the console			[19:23]
<soul9> it's trying to tell you you didn't give it where to ls
<soul9> no, that's warnings
<soul9> if you don't want that do tahoe cmd ... blabla 2> /dev/null
<PovAddict> "KeyError: 0" and "AssertionError" are not warnings, they are
	    exceptions							[19:24]
<soul9> they end in error because there is nothing to ls
<soul9> tahoe has no "root"
<soul9> so you can't do an ls without giving it a default rootcap
<PovAddict> my point is: why does it say "KeyError: 0" instead of "rootcap
	    required"?
<soul9> which is the tahoe "alias"
<PovAddict> is it an argument I'm supposed to pass after 'ls'?		[19:25]
<soul9> well, that's a good point, and you should definitely open a bug if
	there isn't one allready
<PovAddict> "tahoe ls something"?
<soul9> tahoe ls ROOT:cap:thisisasecuretahoeurlwhateverhexblah
<soul9> yeah, it's an tahoe rootcap
<PovAddict> "tahoe ls --help" doesn't say there are required arguments	[19:26]
<soul9> so e.g. you do tahoe mkdir foo
<soul9> gives you a rootcap back
<soul9> then you can save the rootcap as an alias
<soul9> so you don't need to remember the horridly long hash
<soul9> you can give it a name
<soul9> because if you add an tahoe alias
<soul9> then that will be used as default
<PovAddict> okay, actually, --help doesn't even mention any argument	[19:27]
<PovAddict> just the options
<soul9> which is why it's looking for the tahoe key
<PovAddict> and "tahoe <command> [command options] mkdir [options]" makes no
	    sense :) mkdir *is* the <command>
<PovAddict> anyway; what about the rm error?
* soul9 → bugs
<soul9> try it with a rootcap						[19:28]
<soul9> it'll work
<soul9> tahoe is well coded for the purpose it serves
<soul9> wich is uploading big files using the web ui
<soul9> because the cmdline client also just uses the webui in the backend
									[19:29]
<soul9> hell, the fuse client just uses the webui aswell
<PovAddict> I don't know how I'm supposed to get that 'rootcap'
<PovAddict> I uploaded a file from the web ui, now I want to delete it
*** jsgf (n=jeremy@adsl-69-107-65-92.dsl.pltn13.pacbell.net) has quit: Read
    error: 110 (Connection timed out)
<PovAddict> I have the URI I got after uploading			[19:30]
<soul9> that's the cap
<PovAddict> and I can use it to download the file
<soul9> that big hash
<PovAddict> yup
<soul9> so you do tahoe rm HASH
<soul9> tahoe ls HASH
<PovAddict> and it throws an AssertionError, as shown on my paste	[19:31]
<soul9> tahoe cp foo HASH:dor/you/want
<soul9> hm								[19:32]
<soul9> dunno
<soul9> worksforme™
<PovAddict> also note line 16 of paste was wrapped; I *did* pass a hash to ls
<PovAddict> and it just throws a different error			[19:33]
<soul9> then you have a problem in your setup, or the directory is empty
<soul9> iirc
<PovAddict> I didn't create a directory					[19:34]
<PovAddict> I uploaded it from here: http://nooxie.zooko.com:9798/ using the
	    "Upload a file" form, I didn't create a directory first
									[19:35]
<PovAddict> unfortunately none of the shares ended up on my own node, which is
	    kind of what I wanted to try				[19:36]
<PovAddict> so, now, I want to delete the file from the grid; and I only have
	    the one hash it gave me
<soul9> yeah, tahoe ls file doesn't work				[19:38]
<soul9> you can't ls a file
<soul9> use the hash to delete it
<soul9> it's as simple as tahoe rm HASH
<PovAddict> I did and it throws AssertionError
<PovAddict> the hash is in the pastebin; can you try rm yourself?
<soul9> is the file still there tho?					[19:39]
<soul9> i'm not connected to the volounteer grid
<soul9> i have my own ;)
<PovAddict> 'check' says the file is healthy
<PovAddict> Summary: Healthy
<soul9> weird, it should work
<PovAddict>  storage index: xetcmqax6bi5eozzggamxh7gg4			[19:40]
<PovAddict>  good-shares: 10 (encoding is 3-of-10)
<soul9> mmmm								[19:41]
<soul9> º what is the introducer furl to the volounteer grid?		[19:42]
<PovAddict>
	    pb://6cypm5tfsv6ag43g3j5z74qx5nfxa2qq@207.7.145.200:64228,nooxie.zooko.com:64228/introducer
<soul9> mm								[19:45]
<soul9> that's a bug!							[19:46]
<soul9> can't rm it
<soul9> :D
<PovAddict> too bad the contest is about hacking tahoe and not just breaking
	    it :)							[19:47]
<soul9> :D
<soul9> include that hash in the bugreport				[19:49]
<soul9> what version of tahoe are you using
<soul9> ?
<PovAddict> 1.4.1
<PovAddict> I'm currently running it under a debugger, stepping through
	    get_alias							[19:50]
<PovAddict> ok so, if the path starts with "URI: but doesn't contain ":./",
	    then get_alias returns "" for path				[19:51]
<PovAddict> and then 'assert path' explodes
<soul9> really, this is weird man					[19:52]
<soul9> i have put stuff and deleted through the cmdline and the webui and
	mixes
<soul9> never encountered this
<PovAddict> I think I got it						[19:53]
<PovAddict> but I'll upload another file and try again
<PovAddict> 'rm' seems not to be prepared to delete files out of directories...
									[20:03]
<PovAddict> hmm looks like you can't delete files actually!		[20:06]
<PovAddict> "Note that this does not actually delete the file or directory
	    that the name points to from the tahoe grid -- it only removes the
	    named reference from this directory."			[20:07]
<PovAddict> the object will just become "unreachable" (but the data still
	    there?) if no reachable directories have the file, and everybody
	    loses the readcap						[20:08]
<PovAddict> so when do shares actually disappear?			[20:10]
<soul9> i think the crawxler is the garbage collector			[20:14]

[davidsarah]

Hmm. This is how Unix rm has always worked -- it doesn't delete a file if there is another link or open fd to it. The only difference on Tahoe is that since it uses capabilities that have a string representation, there can always be a copy of that representation somewhere.

I'm not sure that renaming it to unlink would help much. The docs could certainly be improved, though -- for example tahoe rm --help says nothing about what rm actually does! Fixing that would be easy.

[zooko]

I'm not sure "This is how unix has always done it." is a sufficient response to the criticism of "Users are confused by it.". :-)

Also, unix traditionally disallows hardlinks to directories (which Tahoe-LAFS doesn't), and also unix users traditionally abjure hardlinks and regard them as mysterious and as trouble waiting to happen.

[warner]

I think part of the problem has to do with how users visualize their filesystems. The notion that a file is "always there", and that a name is an inherent property of the file itself (rather than a path by which you reach the anonymous file), gets in the way.

Maybe our docs need an animated movie.. a first-person perspective that starts with a bunch of swirling blobs of ciphertext (labeled with their storage-index values), then shows your private/aliases file as a book, then shows a readcap in that book, then shows that leading to a ciphertext blob, that gets decrypted and unwraps into a directory table, with filecaps and subdircaps, then you follow a name in that table to the next node, etc. Showing somebody zooming along a link named "foo" to reach a file (that happens to be a picture or a movie or something) might help teach the idea that names are outbound links from directories, rather than inbound properties of their target files/subdirs. And having that concept firmly grounded would make it more clear why "tahoe rm" acts on directories, not files.

Or maybe I'm just looking for another excuse to learn Blender..

[davidsarah]

Replying to zooko:

I'm not sure "This is how unix has always done it." is a sufficient response to the criticism of "Users are confused by it.". :-)

Didn't intend to suggest it was. I do think this is basically a documentation bug rather than a problem with tahoe rm's behaviour, though. (A "destroy" operation as described half-way down ​http://allmydata.org/pipermail/tahoe-dev/2009-September/002861.html and in ​http://allmydata.org/pipermail/tahoe-dev/2009-October/002957.html , in contrast, would act on files, not directory entries.)

How about having rm output a message such as "n directory entries removed (rm does not delete files)."?

[zooko]

David-Sarah: I like your suggestion about the output message!

Brian: I've been teaching Irby unix. I explained that he is in a room, and he can use the following spells: the "look and see" spell, which is performed with the magical incantation "ls", to see what is in the room, the "cross door" spell, incantation "cd" to exit the room through the door with the given sign on it, and the "vigorously interact" spell, incantation "vi" to pick up and use one of the scrolls that are lying around. Also he eventually learned the "copy" spell ("cp").

Therefore, it has always been perfectly intuitive to Irby that rooms don't have signs on them -- doors do -- and that different doors can lead into the same room, and so on. :-)

[davidsarah]

Replying to zooko:

David-Sarah: I like your suggestion about the output message!

Will do. I should have time to work on the bugs assigned to me in the next few days. (Off-topic for this bug, but is there a target date for the 1.6 release?)

Brian: I've been teaching Irby unix.

... and you're also teaching him reference/capability semantics in general :-)

[kevan]

It'd also be nice if this:

nicolas@ubuntu /tmp/tahoe-c$ ~/src/allmydata-tahoe-1.4.1/bin/tahoe rm URI:CHK:2nxmn6eew3y5e5y7ejzwfr2jiq:6mi2hcbpuvkfcttuezi4x3fxwrnmwidql2rtwwtczpccxgz3ez2q:3:10:310946
      Traceback (most recent call last):
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/support/bin/tahoe", line 8, in <module>
          load_entry_point('allmydata-tahoe==1.4.1', 'console_scripts', 'tahoe')()
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/runner.py", line 91, in run
          rc = runner(sys.argv[1:])
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/runner.py", line 78, in runner
          rc = cli.dispatch[command](so)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/cli.py", line 409, in rm
          rc = tahoe_rm.rm(options)
        File "/home/nicolas/src/allmydata-tahoe-1.4.1/src/allmydata/scripts/tahoe_rm.py", line 25, in rm
          assert path
      AssertionError

said something like this:

nicolas@ubuntu /tmp/tahoe-c$ ~/src/allmydata-tahoe-1.4.1/bin/tahoe rm URI:CHK:2nxmn6eew3y5e5y7ejzwfr2jiq:6mi2hcbpuvkfcttuezi4x3fxwrnmwidql2rtwwtczpccxgz3ez2q:3:10:310946
error: 'tahoe rm' can only be used to unlink a file within a directory. Please specify the file that you wish to unlink as alias:path/to/file.

or something similar. Users are rightly confused if our CLI tools output stack traces instead of meaningful error messages; see ​http://allmydata.org/pipermail/tahoe-dev/2010-February/003950.html for a recent example.

(the first exception message should have been fixed in #939; the second in #771)

[davidsarah]

Related to #1104 (the button to unlink a child from a directory should not be labelled "del")

[davidsarah]

CLI: add 'tahoe unlink' as an alias to 'tahoe rm', for forward-compatibility.

[davidsarah]

My suggestion is to add tahoe unlink as an alias in 1.7.1, and then remove tahoe rm in 1.8.0.

(A minor nit with the patch is that tahoe unlink --help gives its synopsis as tahoe rm REMOTE_FILE, but I don't think that matters.)

[zooko]

I don't at this time agree to remove tahoe rm as soon as 1.8.0. However, I reviewed the patch and it looked good.

[davidsarah]

tahoe-unlink-alias.dpatch (actually a slightly different version with changes to NEWS and CREDITS) applied in 809078571610ad93.

[davidsarah]

We can make the documentation and code use 'unlink' for 1.8 (or 1.8beta), but leave tahoe rm as an alias until some future version.

[davidsarah]

Replying to davidsarah:

tahoe-unlink-alias.dpatch (actually a slightly different version with changes to NEWS and CREDITS) applied in 809078571610ad93.

This patch had been manually edited and so applied incorrectly. Fixed in 0d79a4a7d14e4689 (which should refer to this ticket, not #1072).

[davidsarah]

The 'tahoe unlink' alias is in place; doc changes are for 1.9.

[davidsarah]

cleanup: implement rm as a synonym for unlink rather than vice-versa. refs #776 (This depends on the patch for #1359.)

[kevan]

There's a conflict between this patch and trunk: darcs doesn't know what to do about the _create_test_file function in test_cli.Rm.

On my system, test_cli.py tries to import tahoe_rm, which isn't there anymore, and also tries to quiet pyflakes for tahoe_rm. I changed these to tahoe_unlink; without that, the CLI tests don't run.

There's a test in test_cli.Unlink, test_rm_without_path, that still refers to the rm command. I'm not sure if that's an oversight; to me, it seems like it'd be more consistent if it used unlink terminology.

Is it worthwhile to test that tahoe rm behaves like tahoe unlink? Unless I'm mistaken, the only test we have of that is the one that is left with rm terminology in the Unlink class.

The error message on line 26 of tahoe_unlink.py still refers to tahoe rm.

Not particularly related to this ticket, save for the fact that the patches were in the same bundle, but you missed a couple of places when adding the self.command_name value:

• CpOptions in cli.py
• CreateKeyGeneratorOptions in keygen.py
• CreateStatsGathererOptions in stats_gatherer.py

[david-sarah@…]

In 06a5d0c1a3135b8c:

cleanup: implement rm as a synonym for unlink rather than vice-versa. refs #776

[david-sarah@…]

In 9ba8a1b83e3d598e:

docs/frontends/webapi.rst: change some more instances of 'delete' or 'remove' to 'unlink', change some section titles, and use two blank lines between all sections. refs #776, #1104

[david-sarah@…]

In 2da3f69f25645ca6:

Address Kevan's comment in #776 about Options classes missed when adding 'self.command_name'. refs #776, #1359

[davidsarah]

For 1.10, I want to implement the output message suggestion I made in comment:4.

[zooko]

Okay here are all the ideas on this ticket for how to improve this. The ones we've already done are marked with strikethru.

• Make an animated 3D movie showing how to traverse and manipulate a Tahoe-LAFS filesystem (comment:3).
• Teach your children unix/capabilities/reference semantics in terms of a text-based dungeon crawling adventure (comment:5).
• Add a command named tahoe unlink and make it be the primary documented command (comment:13).
• Have tahoe rm or tahoe unlink output a message such as "n directory entries removed (tahoe unlink does not delete files)." (comment:4).
• Make tahoe rm $CAP or tahoe unlink $CAP emit a nice and explanatory error message instead of a mysterious stack trace. (comment:9)
• Remove the tahoe rm command now that tahoe unlink has been the primary documented command for a couple of major releases (comment:13).

The ones that we haven't done yet are still good ideas! Maybe we should open separate tickets for them since they are separately closeable and then people will be able to better remember to do them.

[davidsarah]

Replying to zooko:

Okay here are all the ideas on this ticket for how to improve this. The ones we've already done are marked with strikethru.

[...]

• Make tahoe rm $CAP or tahoe unlink $CAP emit a nice and explanatory error message instead of a mysterious stack trace. (comment:9)

This was fixed in #695 / #1292. The message is

'tahoe unlink' can only unlink directory entries, so a path must be given.

(It says 'tahoe rm' if invoked as that.)

[zooko]

Okay, now, per comment:16, another one is checked off. I'm opening new tickets for the remaining tasks. Remaining to do:

• Make an animated 3D movie showing how to traverse and manipulate a Tahoe-LAFS filesystem (comment:3) (#1826).
• Teach your children unix/capabilities/reference semantics in terms of a text-based dungeon crawling adventure (comment:5).
• Add a command named tahoe unlink and make it be the primary documented command (comment:13).
• Have tahoe rm or tahoe unlink output a message such as "n directory entries removed (tahoe unlink does not delete files)." (comment:4) (#1825).
• Make tahoe rm $CAP or tahoe unlink $CAP emit a nice and explanatory error message instead of a mysterious stack trace. (comment:9)
• Remove the tahoe rm command now that tahoe unlink has been the primary documented command for a couple of major releases (comment:13). (#1827).

[zooko]

I've opened new tickets to hold all the unfinished tasks, closing this one.