integration: should tahoe be a library? or have plugins?

[Brian Warner]

At today's Nuts+Bolts devchat, Meejah and I talked about various ways to
integrate Tahoe with other projects. The inspiration was Least
Authority's "magic-folders" project, which we're busily trying to land
in Tahoe proper (PR#275, maybe landing next week).

(In case you haven't heard of it, "magic-folders" is a two-way directory
synchronization tool, like Dropbox except it has all the end-to-end
encryption that you love and expect from Tahoe. It will replace the
"drop-upload" one-way sync tool that's in the current 1.11 release. Both
are somewhat experimental, and may undergo incompatible changes in the
future).

The next Tahoe release will include magic-folders built-in to the
"tahoe" executable: you'll configure it with a new tahoe.cfg section,
you'll drive it with CLI commands like "tahoe magic-folder create" and
"invite", and you can check on its status through the regular Tahoe web
interface. The service starts an inotify watcher at node startup, and
reacts to local file changes by uploading data to a shared folder, and
reacts to changes from other clients by downloading data to the local
disk.

We're looking forward to landing this and making magic-folders available
to all Tahoe users, but there are some potential drawbacks to putting it
in Tahoe directly:

* if there are any new dependencies, all users must install them even if
  they don't use the new feature
* unit tests take longer to run, to exercise the new feature
* merge conflicts make long-lived branches hard to develop/maintain

We have other tahoe-related projects with similar issues:

* "tahoe backup" really wants to be a long-running daemon (to run
  periodically without danger of overlap, and to react to changed files)
* we'd like to build a "repair agent", which periodically checks up on
  your files and repairs any that are missing shares
* the FTP/SFTP frontends are not used by everybody, and add dependencies
* heck, the human-facing WUI (web UI) isn't used by everybody
* Grid-Sync is a standalone application, but needs a Tahoe agent

There are at least two other ways we might deliver this. The first is by
shipping magic-folders (or any of these projects) as a separate
application on *top* of Tahoe. The application might make RPC calls to
an external Tahoe process (like apps which use MySQL or Tor), or it
might import Tahoe as a library (like apps which use SQLite). Many of
these applications only need upload/download functionality, which takes
less configuration, and might be easier to shrink down into something
resembling a library. This would also tighten up Tahoe's abstraction
boundaries: encryption / encoding / share-placement are layered features
that live underneath an application, and on top of a network, and each
layer might be useful independently (and could be swapped out for an
alternate implementation)

In that world, the standard Tahoe node might shrink down to just an
uploader/downloader and a machine-oriented web API port. Or a library
that would be imported independently by each app. The filesystem browser
could be shipped as a separate application, as would the storage server.

The second approach would be to turn magic-folder into a plugin, that
lives *under* the "one true Tahoe process". You'd add a clause to
tahoe.cfg to enable+configure it, and the right services would be
started, and statuses rendered, but the core Tahoe code wouldn't know
anything about magic-folders directly. It would just provide the right
plugin interfaces (one to register webapi calls, another to render new
status URLs, a third to launch a Service at node startup for
periodic/background tasks).

In this world, the Tahoe node would shrink, but not as much: we'd move
some of the functionality out into enabled-by-default plugins, and some
pieces might move into other repos entirely, but we'd still need a core
plugin-container. The difficult merge conflicts would turn into
difficult-to-maintain API contracts (inability to make some improvement
because there are plugins that depend upon the old API, etc).

Testing would be split up. Plugins would run their own CI tests against
the current-release and current-trunk of Tahoe. We'd want something to
tell us if a change to core Tahoe broke some plugin.


Anyways, I know some folks here have built projects that use Tahoe in
some way.. does anyone have thoughts about any of these approaches?

thanks,
 -Brian



_ PR#275: https://github.com/tahoe-lafs/tahoe-lafs/pull/275