[tahoe-lafs-trac-stream] [Tahoe-LAFS] #3985: Support "storage plugins" in the Great Black Swamp storage server and client

[Tahoe-LAFS]

#3985: Support "storage plugins" in the Great Black Swamp storage server and
client
-------------------------+-----------------------
     Reporter:  exarkun  |      Owner:
         Type:  defect   |     Status:  new
     Priority:  normal   |  Milestone:  undecided
    Component:  unknown  |    Version:  n/a
   Resolution:           |   Keywords:
Launchpad Bug:           |
-------------------------+-----------------------

Comment (by exarkun):

 An obvious approach is to transliterate the existing plugin interface to
 Great Black Swamp and allow third-parties to implement a new plugin for
 the new interface.  The plugins should have comparable functionality
 because Great Black Swamp doesn't change much about the _semantics_ of the
 storage protocol, just the network layer.

 A entirely different approach is to leverage the _network layer_ *as* the
 plugin interface and have third-parties provide this functionality out-of-
 process instead of as loadable Python modules to execute in-process.

 Whereas the former process (for clients) looks something like:

 * Load server configuration
 * Load plugin code
 * Match each server to either the built-in storage client or one of the
 loaded plugins
 * Use the resulting object to make storage API calls on the server

 The process for the latter idea (for clients) would look something like:

 * Load server configuration
 * Match each server using the built-in storage client
 * Use the resulting object to make storage API calls to the server

 *plus* a separate third-party workflow that looks something like:

 * Provision a NURL for a locally-running Great Black Swamp "plugin"
 process
 * Give the storage client node server configuration pointing at this
   (To give the client node N storage servers, provision N NURLs for the
 "plugin" process)
 * Implement plugin-specific logic in the "plugin" process
   (Probably making desired changes and then proxying most operations
 onwards to some set of real servers)

 Advantages of this out-of-process approach:

 * Leave the existing Tahoe-LAFS Python code alone.
   * There is substantial complexity in the current plugin implementation
 to deal with the quirks of the way a Tahoe-LAFS client node is
 representation.
     This could all be avoided.
 * Plugins can be implemented in any language.
   * This might save the work of having to implement each plugin more than
 once.
     The only known implementation already wants to support a Haskell
 client.

 Disadvantages:

 * An extra process to manage.
 * Integration with dynamic server discovery (ie, introducers) involves
 some more work.
   * The introducer protocol is still Foolscap (though we already knew we
 wanted to change that)

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3985#comment:1>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage