command-line interface

[zooko]

I met a nice guy named Jeff Hehner at Caffe Sole tonight, who said that he wanted a command-line interface to a tool like Tahoe.

I agreed with him that command-line interfaces are very good to have.

I'm marking this one as "priority: major". Along with the XML-RPC interface (#48) this might become my next primary focus. Basically, Tahoe already works well enough for several valuable use cases, and now we just need to enable people to use it. (Meaning, of course, people who are enabled by CLI and XML-RPC... People who are enabled by Windows filesystem integration come next...)

This is already mentioned in source:roadmap.txt .

[zooko]

I'm motivated to work on this. The first question is: does the process that you execute block until the operation completes or do we do some funky trick such as it producing a randomly-named temporary file that it later populates when the operation completes? Tahoe (and all of our thinking) is strongly in the non-blocking, event-based paradigm, but I think this is an exception. The purpose of this component is to interface with Unix users and programmers, and Unix users and programmers expect their long-running command-lines to block.

Raising the priority, not because it is "critical", but to indicate that I'm more motivated by this than by some other tickets. Actually perhaps it is "critical". A word like "critical" begs the question "critical for *what*?", and this ticket is critical for getting certain Unix system administrators to give tahoe a spin.

[warner]

I think the CLI process should block until the operation completes. I'd like to see a 'tahoe put LOCALFILE VDRIVEFILENAME', 'tahoe get VDRIVEFILENAME LOCALFILE', and then a collection of mv/rm/cp type operations. Actually an interface like scp's would be great (which would imply 'tahoe-cp ~/stuff.txt vdrive:here.txt', or something).

I'm assuming that there is a long-running daemon, and that the CLI tool is a short-lived process that connects to the daemon, tells it to do something, waits for it to indicate completion, emits a message, then quits. If this is the case, both Foolscap and XMLRPC are useful approaches (with either a FURL or a webfront-style "account handle" as the secret authentication bootstrapping pieces). Another technique I've used the past (on buildbot) is to have a special "command directory", like an inbox, which is a Maildir where you can drop in specially-formatted uniquely-named files. A file might have "upload" on the first line, a local filename on the second line, and a vdrive filename on the third line. The daemon could poll this directory every few seconds (or use inotify), execute the commands it finds there, and delete the file when it is done. This reduces the authentication-bootstrap issue to writability of a specific directory, and doesn't require any network for the CLI tool at all.

(incidentally, I'm unsure about using trac's 'priority' field to indicate which tickets we're currently interested in.. we aren't confined to resolving these tickets in any particular order. maybe we need another field named 'coolness' or 'sexyness' for this purpose..)

[zooko]

Re: trac "priority" field, let's discuss it in #60.

Re: scp interface: Perfect! Thanks.

Re: command-line backend -- how about if we implement #48 -- XML-RPC interface -- and then use that for the command-line client?

[warner]

in thinking about this more, I think I want to see the client provide a virtual drive (composed of a variety of filetrees) in exactly the same way that a kernel provides a single filesystem (composed of a variety of volumes) to applications. The basic ls/cp/rm/mkdir interface can treat this vdrive as a single filesystem, without being too aware of the pieces that it comprises.

More advanced tools can use additional APIs which expose the filetree pieces, and provide ways to manipulate them (splitting a single filetree into multiple pieces so one can be shared without the other, creating a "virtual CD" snapshot from a given root, etc). But frontends that want to think of the vdrive as just a regular filesystem are free to do so.

[warner]

robk rightly points out that we should look at xuilcli (from amdlib), and see if we can adapt that.

[warner]

we have 'tahoe ls' and 'tahoe get' working right now. I think we should declare our 0.5.0 goals to be 'tahoe get -r' and 'tahoe put -r' as well, and then call it a release. Maybe add 'rm' as well. But 'cp' and 'mv' can probably wait for 0.5.1 .

[zooko]

This is a blocker for the v0.5 release.

[zooko]

Let's make it so that "allmydata-tahoe ls foo/bar.txt" lists bar.txt (if it exists). Currently ls ignores arguments which aren't directories.

[zooko]

Nathan had a good question: why do we have the "-r" flag? If someone asks to "tahoe get" a directory, then what else could they mean other than that they want to recursively get the directory's contents?

[warner]

I suspect the usual answer is that there's a significant probability that they referred to a directory by mistake, and might be surprised to find themselves downloading hundreds of files instead of just one.

A weaker answer is "because all the other tools do it", and consistency has some merits.

And the historical answer is probably that you can actually open and read directories, although the results are not normally useful, and I think more recent unix kernels removed this ability.

[zooko]

Good enough for me.

[zooko]

Interesting, "tahoe get ADIRECTORY" currently emits an html page describing that directory. :-)

[zooko]

Okay, so far we have:

allmydata-tahoe ls
allmydata-tahoe get

We would like to add

allmydata-tahoe put

And that will be good enough for v0.5! (Note that we won't yet support "-r", nor a bunch of features for "ls", etc.)

[zooko]

Almost done with "put"...

[zooko]

Fixing some UI issues concerning --node-url.

Probably going to add "rm" real quick...

[zooko]

Implemented rm. 0c22044f7edb16b4

Now doing extra manual testing before v0.5 release!