separate "gateway state directory" from "client state directory"

[zooko]

updating the Description for clarity:

This ticket is about the proposal to have separate directories for holding the state/configuration of the LAFS gateway from the state/configuration of the LAFS client. In the current (Tahoe-LAFS v1.10) code, both of those things are maintained in one shared directory, called the "node directory" or "base directory". The state therein is actually non-overlapping:

• things a client (i.e. the "tahoe" command-line tool) uses out of the base directory:
  • the node.url file to find out how to connect to the gateway
  • the private/backupdb.sqlite file to find out what files have already been uploaded
  • the private/aliases file holding aliases to caps

• things a gateway uses out of the base directory:
  • everything else that is stored in there except for the node.url, private/aliases and private/backupdb.sqlite files

So the client never uses any of the files that are kept in that directory for the gateway's purposes, and the gateway never uses any of the files that are kept in that directory for the client's purposes.

---

I use multiple grids (pub grid, volunteergrid, and a private family grid), and I just now had a confusing error where I ran tahoe backup and it completed quickly but produced a backup directory full of links to files with 0 shares each.

What happened, of course, was that I had previously run tahoe backup --node-url=http://127.0.0.1:3458/ to backup these files to my family grid, and now I was running tahoe backup --node-url=http://127.0.0.1:3457/ to backup these files to the volunteergrid, but I was unwittingly using the same backupdb.sqlite.

I wonder if, when the --node-url option is present, then the CLI shouldn't look into ~/.tahoe at all. Most of the configuration and state in ~/.tahoe is specific to the gateway that the --node-url points to, and the CLI will ignore it anyway and instead whatever configuration is in the tahoe-base-dir that is used by the gateway will take effect.

The only exception that I can think of right away is the private/backupdb.sqlite. Is that the only thing that affects the CLI when --node-url is present? Maybe it should be kept in a different directory.

I think I'm a bit confused about this. I'm not sure what all it means that there exists a ~/.tahoe when I'm actually using a gateway which runs as a separate user process, is specified by the --node-url option, and it has its own ~/.tahoe in its own user account. As a work-around and a way to gain clarity, I'll probably start specifying --node-directory in addition to --node-url, but this really feels wrong as it isn't a node directory at all! It is a CLI directory. :-)

[davidsarah]

Duplicate of #977.

I think I'm a bit confused about this. I'm not sure what all it means that there exists a ~/.tahoe when I'm actually using a gateway which runs as a separate user process, is specified by the --node-url option, and it has its own ~/.tahoe in its own user account.

I also found that confusing.

[zooko]

I'm not sure that I agree that this is just a duplicate of #977 (backupdb should store which grid it is scoped to). What about the parts of ~/.tahoe which don't have to do with backup at all? It is still confusing that the command

tahoe put --node-url=http://127.0.0.1:3456 --node-directory=~/.tahoe-volunteergrid MYFILE

will not use any of your configuration from your ~/.tahoe-volunteergrid directory and indeed may not touch the grid you call "volunteergrid" at all.

[zooko]

Hm, I'm not sure about this, but one potential solution to this issue would be for there to be a --cli-directory= option which points to a directory that contains nothing but an optional backupdb and a node URL. Hm, in fact that's all that the cli ever uses the so-called --node-directory for, right? It is really not true that you ever tell the CLI what the node directory is. You really tell it where to find the node URL and then it contacts a node with that URL and that node uses whatever node directory it is configured to use. So the option to the CLI named --node-directory is a misnomer.

[mlakewood]

In addition to this i've experienced the following problem. If I create a client node in the default directory ie .tahoe through tahoe create-client command, and then modify the web.port = tcp:3456:interface=127.0.0.1 to be say web.port = tcp:3456:interface=10.0.10.89 when you try to run a command it will thow and exception about trying to connect to the client. This is because in node.url its still pointing at ​http://127.0.0.1:3456 . If you change this to point at 10.0.10.89:3456 then all is good. however if you stop and start the tahoe client then the node.url is reset to ​http://127.0.0.1:3456 . which is a bit of a usability problem.. node.url seems like a configuration file, but gets overridden with a potentially invalid default. It seems to me node.url shouldn't be overriden on tahoe start.

[edited to make all the examples use the same port number, which I believe is what mlakewood intended --Zooko]

[zooko]

Straw man proposal (I'm sleepy!):

Remove --node-directory from the CLI and document the fact that users can write tahoe --node-url=`cat ~/.tahoe/node.url` ls $DIRCAP if they wish. This syntax will make it clear to the user who inspects it that the tahoe command is not reading the contents of ~/.tahoe directory aside from the contents of that one file.

[warner]

CLI tools look in --node-directory for things that are generated by:

• the tahoe client node, to help the CLI tools find the node
• the CLI tools themselves, that they want to retrieve later

At present, the only from-node-to-CLI things are node.url, to find the webapi port. In the future, I'm considering adding a few more files, starting with a $NODEDIR/private/control.key which would help a tahoe webopen --control-panel derive a secret URL that points to node-control functions, outside the scope of any particular filecap. Another possibility is an accounting.secret, which would let the CLI tools create webapi URLs that include authority to use space on a grid (rather than the ambient authority that currently grants storage rights to anyone who can talk to the webapi port). Also, $NODEDIR/private/control.furl could be used by CLI commands that wanted foolscap-based access (which would be easier for some purposes than HTTP-based access).

The set of from-CLI-to-CLI things in there currently includes:

• private/aliases: allows "tahoe ls home:" instead of "tahoe ls URI:DIR2:blahblahblahblahblah"
• private/backupdb.sqlite: used by 'tahoe backup', potentially 'tahoe cp'

I named it --node-directory because you can set it equal to a tahoe client node's base directory (which is what I usually mean by "$NODEDIR") and then the CLI tool will correctly find everything it needs to work without any extra effort on your part (and the secret things that it writes will go into a well-named directory that's already chmod go-rwx). And, because --node-directory is consistent across all tahoe commands, including tahoe create-node and tahoe start. So you could make a one-line shell script which did tahoe --node-directory=~/.other $* named othertahoe and use that for everything and it would Just Work (well, if we fixed the option parser to look for --node-directory before the command instead of after).

I've gotta run, I'll come back to this ticket later to chime in about the proposal. At first glance, --cli-directory does sound more accurate, but I think the parallelism of --node-directory everywhere is valuable.

[zooko]

I still have the feeling that tahoe-lafs gateways and tahoe-lafs clients are separate objects (for example, I prefer to run my gateways and my clients under separate user accounts on the same machine sometimes), and they should not share state in this way. I find it confusing.

Replying to warner:

I've gotta run, I'll come back to this ticket later to chime in about the proposal. At first glance, --cli-directory does sound more accurate, but I think the parallelism of --node-directory everywhere is valuable.

I have the feeling that this is really wrong -- that you shouldn't say --node-directory to a gateway to mean the directory where it stores its persistent state such as the node.pem and also say --node-directory to a client to mean the directory where it stores its persistent state such as the backupdb.sqlite. It is the "bad" kind of parallelism, when the underlying thing is different, so the interface to it ought to be different too. :-)

(Also, I think it would help us all think and communicate more precisely if we stopped using the ambiguous word "node"... Except in those cases where we actually mean a Tahoe-LAFS process which runs more than one of (server, gateway, introducer) or which runs an unspecified service.)

[gdt]

I broadly concur with zooko in comment 7. Sort of related, as I've been setting up storage nodes in a private grid, I find myself wishing for the grid parameters to be in a separate file from the local parameters, so that I could copy the grid-params.conf file around and just drop it in, and not have that overwrite the node id.

Beyond this, I think that probably one should have a directory to represent client access, and that would then have the aliases file, the backupdb, and a pointer to the WAPI. The gateway node directory would then be separate. Once this is done, the CLI probably should lose the ability to specify the WAPI URL directly, and only point to client directories.

[zooko]

I'm thinking about configuration right now, so I'm interested in this ticket. Commenting here to remind myself about it and to let everyone know that I might get around to working on it sometime.

[daira]

I wouldn't be opposed to adding --cli-directory, but I would be opposed to not having it default to the same thing as --node-directory. Not doing that would break people's existing setups and would be a serious usability regression.

[amontero]

Added private/aliases to client-side files in issue description.

[warner]

(circling back to this ticket thanks to zooko's link from #2045, which is about larger-scale changes to the node's and code's directory layout)

Rereading zooko's initial issue, I found myself tempted to yell out "don't do that!". I guess I've always optimized tahoe's frontend- and setup- management tools for the common case of a single "gateway" per (user*computer) tuple. I really want the instructions to be as simple as "tahoe create; tahoe start; tahoe webopen". I don't want to complicate that for the sake of the less-common use case of multiple nodes/gateways/clients/whatevers.

Partly that indicates a lack of universality in our design (which we've always known about, and always regretted, but also know better than to try and fix, because it's very hard, certainly distracting, probably confusing, and slightly impossible). There's no "one true grid" (#2009). Grids can't be too big (#235, #444), increasing the demand for using multiple ones, in particular if you want to use tahoe to share files with other people. There are no "grid identifiers" in filecaps (#403), so if you want to use multiple distinct grids, you need multiple distinct client nodes ("gateways" in zooko's lexicon) and must be careful to give the right filecap to the right node (WAPI port / gateway process).

A side-note on terminology mismatches: I think (and talk) about tahoe in terms of three pieces:

• 1: frontends (CLI scripts, web browsers, FTP/SFTP clients), all talking over a network connection (WAPI or others) to the client node
• 2: client nodes, which respond to WAPI requests, perform the upload/download/encode/decode algorithms and make connections to server nodes
• 3: server nodes, which are (so far) agnostic about file-encoding formats and just respond to PUT/GET-share requests from client nodes

I think Zooko thinks/talks in the same three pieces but with different names:

• 1: clients (CLI scripts, web browsers, FTP/SFTP clients)
• 2: gateways
• 3: servers

A related angle is the imperfect distinction between the functions performed by pieces 1 and 2. tahoe backup is a good example: this is currently a CLI command, but I feel that it should really be moved into the client node (#1018). Backup is more of an ongoing process than a one-off action (#643). A one-shot CLI command needs to be run from cron to make it into a process, and then it doesn't have enough information to coordinate with other (overlapping) runs (#2062, #2053). I'd like to have backups be managed through some sort of control panel (#1588, #1587), where you can express your priorities and preferences about what you want to be backed up and how much network/CPU it's allowed to consume, and then the backup agent handles the rest. This control panel should also be a place to check in on the process, especially for progress reports during the long initial upload.

There are two big blockers for this sort of long-running agent. The first is how/whether to split this from the long-running code that knows how to upload/download files. We had a good discussion about this at the 2011 summit. Zooko's mental model, which uses the word "gateway", helps make this a bit more clear: my desired backup-manager process would live in an "Agent", and the upload/download stuff would live in the "Gateway", and maybe the Agent would use te Gateway but not vice-versa. Depending upon the value of caching server connections and generally having a long-term relationship with servers (tracking uptime/speed/reliability), it might even make sense for the Gateway functionality to *not* live in a long-term process, and instead be a short-lifetime library that gets loaded on demand (imagine if "tahoe put" were standalone, maybe learning about cached server information from a sqlite database, but establishing its own server connections as necessary).

The other is how to safely talk to this agent, honoring our objcap no-ambient-authority style: some sort of restricted-access web-based control panel (#674, wiki:Summit2Day2#ControlPanel) which I've been prototyping externally in my "toolbed" and "petmail" projects (but it's very JS-heavy).

Anyways, that was a long diversion away from the main point: the use of a single NODEDIR to manage the states and configurations of all these pieces (client-ish stuff, agent-ish stuff, gateway-ish stuff, heck even server-ish stuff) is ideal for one-grid cases, and confusing for multiple-grid cases.

I'm warming slightly to the --cli-directory= idea. Maybe by splitting these different bits of functionality into separate subdirs, putting all of them in the single NODEDIR by default, but making it clear that e.g. CLI commands only touch stuff in NODEDIR/cli/* . Then make it possible to either override the top-level --nodedir=, or a CLI-functionality-specific --cli-directory=.

It's worth remembering the overlap between these components, though. The gateway writes out a node.url file, the frontend commands read it. Control panels will involve access keys being read or written to agent-accessible databases. We might be able to statically construct enough of this that we don't need to think about ongoing config-directory-based communication between components after initial tahoe create, but maybe not.

But I still think it may be easier to tell people "don't do that" and try to make the one-grid-per-user case work better.

[zooko]

Replying to warner:

Rereading zooko's initial issue, I found myself tempted to yell out "don't do that!". I guess I've always optimized tahoe's frontend- and setup- management tools for the common case of a single "gateway" per (user*computer) tuple. I really want the instructions to be as simple as "tahoe create; tahoe start; tahoe webopen". I don't want to complicate that for the sake of the less-common use case of multiple nodes/gateways/clients/whatevers.

…

Anyways, that was a long diversion away from the main point: the use of a single NODEDIR to manage the states and configurations of all these pieces (client-ish stuff, agent-ish stuff, gateway-ish stuff, heck even server-ish stuff) is ideal for one-grid cases, and confusing for multiple-grid cases.

I'm not sure if the "separate dirs for separate state" approach is any more complicated, at least for the end user.

Specifically, for the proposal Brian suggests here:

I'm warming slightly to the --cli-directory= idea. Maybe by splitting these different bits of functionality into separate subdirs, putting all of them in the single NODEDIR by default, but making it clear that e.g. CLI commands only touch stuff in NODEDIR/cli/* . Then make it possible to either override the top-level --nodedir=, or a CLI-functionality-specific --cli-directory=.

What impact would this have on the end-user-visible CLI flow, such as the tahoe create; tahoe start; tahoe webopen that Brian mentioned?

[warner]

Hm. Ok, imagine this (I'm not sure I like it yet, but let's think about it):

• emphasize well-defined boundaries of responsibility in non-server-side docs: "Gateway Process", "Agent Process", "CLI Tools" (oneshot)
• users must create/configure the three things separately
• first: tahoe create-gateway GATEWAYDIR. This directory holds human-edited config files and machine-edited state files, but only for gateway-related things: cached introducer data, server connection data, historical server reliability, etc. Maybe cached ciphertext (download caching, or upload caching to satisfy Shawn Willden's request for faster backup-and-shutdown handoff of responsibility from Agent to Gateway)
• second: tahoe create-agent AGENTDIR GATEWAYDIR, where the AGENTDIR holds agent-related state, and the GATEWAYDIR is only used to figure out how to contact the gateway. Maybe tahoe create-agent AGENTDIR GATEWAYURL instead (except does it need to access anything else?).
• third: tahoe configure-clients CLIDIR GATEWAYURL.

Except:

• how do we want to split responsibility between CLI (client) scripts and the gateway? Like, where should the encoding parameters be? When we add accounting, which side holds the key?
• what about CLI scripts that are supposed to talk to the Agent instead of the Gateway? tahoe configure-backup?

I kind of expect that tripling the number of commands that people have to run would roughly triple the perceived level of complexity, even if it exposes distinct components as separate conceptual entities in a way that's less confusing than conflating them.