integration: should tahoe be a library? or have plugins?

Sat May 7 09:21:24 UTC 2016

Hi Warner,

good to see interesting stuff going on with Tahoe!

Just some random two cents from a user and list lurker without thinking
deeply about implications etc:

Intuitively, I preferred the Tahoe-library style because it is probably
way more convenient for plugin developers to use.
Also, for users, it is probably preferable to start just one program
instead of n and possibly struggling to wire them up etc.

However, you point that plugins using Tahoe as a library can lead to
stagnation as a side-effect of maintaining compatibility for them is
striking!

It might be an option to provide a designated Tahoe API in Python that
carries a version number. I cannot estimate how much implementation
effort this implies or how well those architectures work with Twisted.

While writing this email, I feel it might be the most flexible approach
to have a small and neatly designed API that can be used from external
plugins, possibly written in different programming languages.
A (versioned) RESTful Web API is probably most widely accepted nowadays
and people know how to setup HTTP proxies etc. There could be a Python
client for it! :)
This approach also enables administrators to distribute plugins among
multiple machines if they want.

Picking up a point from above, the default Tahoe installation should
stay as simple as possible (understanding the encoding parameters is
probably enough).

Say, I want to use Tahoe with the public grid and the WUI, I could
imagine the following procedure:
#. pip-installing 'tahoe-lafs-wui' (pulls in 'tahoe-lafs')
#. `tahoe create-node`
   (stdout tells me to now edit tahoe.cfg)
#. add nickname, introducer in a section in tahoe.cfg
   clearly highlighted as minimal set of options to configure
#. [optionally, #2382: `tahoe-wui hook`
   which installs a hook in the default node directory that does
   `tahoe-wui start` when the node is started]
#. `tahoe start`
#. [`tahoe-wui start`]

Or just `tahoe quickstart` interactively ;)

Also, the Web API might become even more powerful if the Tahoe devs
/have to/ use it themselves for all kinds of fancy plugins like magic
folders.

Good work folks, keep going for this wonderful project!

Cheers,

Lukas

On 05/06/2016 11:23 PM, Brian Warner wrote as excerpted:
> At today's Nuts+Bolts devchat, Meejah and I talked about various ways to
> integrate Tahoe with other projects. The inspiration was Least
> Authority's "magic-folders" project, which we're busily trying to land
> in Tahoe proper (PR#275, maybe landing next week).
> 
> (In case you haven't heard of it, "magic-folders" is a two-way directory
> synchronization tool, like Dropbox except it has all the end-to-end
> encryption that you love and expect from Tahoe. It will replace the
> "drop-upload" one-way sync tool that's in the current 1.11 release. Both
> are somewhat experimental, and may undergo incompatible changes in the
> future).
> 
> The next Tahoe release will include magic-folders built-in to the
> "tahoe" executable: you'll configure it with a new tahoe.cfg section,
> you'll drive it with CLI commands like "tahoe magic-folder create" and
> "invite", and you can check on its status through the regular Tahoe web
> interface. The service starts an inotify watcher at node startup, and
> reacts to local file changes by uploading data to a shared folder, and
> reacts to changes from other clients by downloading data to the local
> disk.
> 
> We're looking forward to landing this and making magic-folders available
> to all Tahoe users, but there are some potential drawbacks to putting it
> in Tahoe directly:
> 
> * if there are any new dependencies, all users must install them even if
>   they don't use the new feature
> * unit tests take longer to run, to exercise the new feature
> * merge conflicts make long-lived branches hard to develop/maintain
> 
> We have other tahoe-related projects with similar issues:
> 
> * "tahoe backup" really wants to be a long-running daemon (to run
>   periodically without danger of overlap, and to react to changed files)
> * we'd like to build a "repair agent", which periodically checks up on
>   your files and repairs any that are missing shares
> * the FTP/SFTP frontends are not used by everybody, and add dependencies
> * heck, the human-facing WUI (web UI) isn't used by everybody
> * Grid-Sync is a standalone application, but needs a Tahoe agent
> 
> There are at least two other ways we might deliver this. The first is by
> shipping magic-folders (or any of these projects) as a separate
> application on *top* of Tahoe. The application might make RPC calls to
> an external Tahoe process (like apps which use MySQL or Tor), or it
> might import Tahoe as a library (like apps which use SQLite). Many of
> these applications only need upload/download functionality, which takes
> less configuration, and might be easier to shrink down into something
> resembling a library. This would also tighten up Tahoe's abstraction
> boundaries: encryption / encoding / share-placement are layered features
> that live underneath an application, and on top of a network, and each
> layer might be useful independently (and could be swapped out for an
> alternate implementation)
> 
> In that world, the standard Tahoe node might shrink down to just an
> uploader/downloader and a machine-oriented web API port. Or a library
> that would be imported independently by each app. The filesystem browser
> could be shipped as a separate application, as would the storage server.
> 
> The second approach would be to turn magic-folder into a plugin, that
> lives *under* the "one true Tahoe process". You'd add a clause to
> tahoe.cfg to enable+configure it, and the right services would be
> started, and statuses rendered, but the core Tahoe code wouldn't know
> anything about magic-folders directly. It would just provide the right
> plugin interfaces (one to register webapi calls, another to render new
> status URLs, a third to launch a Service at node startup for
> periodic/background tasks).
> 
> In this world, the Tahoe node would shrink, but not as much: we'd move
> some of the functionality out into enabled-by-default plugins, and some
> pieces might move into other repos entirely, but we'd still need a core
> plugin-container. The difficult merge conflicts would turn into
> difficult-to-maintain API contracts (inability to make some improvement
> because there are plugins that depend upon the old API, etc).
> 
> Testing would be split up. Plugins would run their own CI tests against
> the current-release and current-trunk of Tahoe. We'd want something to
> tell us if a change to core Tahoe broke some plugin.
> 
> 
> Anyways, I know some folks here have built projects that use Tahoe in
> some way.. does anyone have thoughts about any of these approaches?
> 
> thanks,
>  -Brian
> 
> 
> 
> _ PR#275: https://github.com/tahoe-lafs/tahoe-lafs/pull/275
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
> 