devchat notes 05-Sep-2017

[Brian Warner]

Tahoe-LAFS devchat 05-Sep-2017

Attendees: warner, exarkun, meejah, cypher, dawuud, bookchin, str4d

Topics:
* exarkun's profiling efforts on the storage server: still trying to get
  it working
* warner's old atlasperf and tahoeperf
* moving away from foolscap, all the same blockers as before
* porting everything to python3
  * cpython 2.7 EOL is 2020
  * pypy plans to support py2 for longer
  * do we run on pypy? zfec/pycryptopp binary deps?
  * switch from pycryptopp to Cryptography
    * maybe add stuff to Cryptography
    * Cryptography is better maintained by far, but uses OpenSSL as the
      backend which we aren't fond of
    * maybe contract for a better Cryptography backend
  * port zfec to py3? or replace with pure-python? or rust-binding?
    * pypy may make the slowdown better
  * probably need to port foolscap to py3 even though we hope to move to
    HTTP
* changing to rainhill: could we drop support for the old encoding
  scheme?
  * str4d suggests to use the share-maintenance (renewal) tools to
    migrate shares to the new format
  * also replace the rootcap with a new-format mutable directory
  * or a "cp -r --upgrade-format" command
  * need a readcap to upgrade the files, since the encoding is so very
    different
    * if it were more similar, a verifycap might suffice, and we could
      delegate the conversion process to someone
    * we'd have to stick with the same encryption key, so sticking with
      AES, and giving up on 100-year-crypto
* switching from foolscap to http could be done incrementally
  * all nodes must enter the "foolscap+http box" before any are allowed
    to leave it (upgrade to http-only)
* switching config from (tahoe.cfg, yaml files) to single sqlite DB
  * warner picked this up from glyph years ago
  * would need user tools to modify config ("tahoe config set")
    * or web-based configurator
  * exarkun points out: user-vs-node modify conflicts still require
    management, doesn't come free
  * storj approach (notes from bookchin):
    * electron app, could modify the config file form within the GUI app
    * used a flat JSON file, worried about coherence, crashes during
      write, trailing commas
    * in new project ("orc"?) can use web login interface to stop child
      nodes, update config, restart nodes
    * bookchin isn't sure he likes it yet
    * less-technical users were more reluctant to manage a config file,
      was better to modify through the GUI
  * exarkun prefers command-line arguments (argv)
  * str4d said i2p had the opposite problem: the web UI is the primary
    config mechanism, config is saved as serialized java properties(?)
    * editing the config file is hard, doesn't retain comments
    * hard to discover the existence of config settings
  * might be one-or-the-other
    * 1: text-based human-editable never-machine-modified config file
    * 2: binary non-human-editable always-machine-modified config file
      (sqlite, provide config-editing tools)
  * backing up the config file, store it in version-control,
    machine-generated config for fleet deployments
  * meejah really likes ephemeral config: no state on disk
    * tor, bitcoin, zcash do this
  * if tahoe client was a library (no server, no magic-folder), this
    would be easier
    * meejah: could use 'tahoe' python package namespace to greenfield
      this API
    * the client() constructor could take the basic args
      * c = client(introducer_furl)
      * c.put(data, encoding_parameters=) -> filecap
      * c.get(filecap) -> data
    * or without a constructor: tahoe.client.put(introducer_furl, data,
      encoding_parameters)
      * but that makes it harder to cache data or connections
    * if it needs any persistence, pass *in* objects to provide that
      * twisted.conch does this: privkey is maintained in an object that
        is passed into the API
      * txtorcon too: pass in a function that can be called to get a
        password (via Deferred)
        * callers can provide a static password, or ask the user
    * applications could decide whether the persistence is important or
      not, pass in objects to provide it
      * cached server records
      * keepalive-d server connections
    * "tahoe backup" is an application that manages its own persistent
      state (backupdb)
    * related client and server instances (reciprocity accounting)
      * add an Accounting object that is passed into the constructor?
    * where should the webstatus/progress info go?
      * could pass a web-UI monitoring object into the API
      * normal filecap=c.put(data) probably wants progress data too
      * status of specific operation
      * status of all operations, all client connections
* could split tahoe into Client and Server objects (interfaces)
  * also split allmydata.web.* into client pieces and server pieces
  * client daemon gets rewritten in terms of a non-daemonic client
    libtahoe API
  * applications can embed the client libtahoe API
* meejah would like to see server split up too: make a function that
  accepts objects for config/state/accounting-db, build server daemon
  out of that
* pass reactor, treq.Agent, everything powerful *into* the client
  constructor/factory function?