devchat notes from 24-Jan-2017

[Brian Warner]

Notes for tahoe devchat 24-Jan-2017

attendees: ramki, warner, meejah, exarkun, dawuud, cypher

* 1.12.1 landed in debian unstable, will migrate to testing 29-jan (with
  foolscap)
 * https://tracker.debian.org/pkg/tahoe-lafs
 * "tahoe --version" has (cosmetic) warnings about missing dependencies
 * artifact of debian's installed packages using .egg-info/ with empty
   requires.txt

* #1382 (servers-of-happiness): dawuud and meejah have been refactoring
  the branch, improving the tests
 * made small change to spec, server with no space is treated as
   read-only
 * the actual spec-change is in 17c562129d464e000a3a7e0d14b4d751bf3be0e6
 * In step 6, "Let an edge exist between server S and share T if and
   only if S already has T, or *could* hold T (i.e. S has enough
   available space to hold a share of at least T's size)." -> becomes
   "Construct a bipartite graph G3 of (only readwrite) servers to shares
   (some shares may already exist on a server)."
 * there's some code duplication that could be refactored, maybe in the
   future, in util/happinessutil.py

* #2861 (I2P vs TLS) is still broken, warner needs some time with str4d
  and wireshark to debug

* #2476 JSON welcome page: dawuud is getting ready to land
 * provides several pieces: introducer info, other-servers info,
   my-storage-server status info, versions
 * should we make separate child URL paths for the individual pieces?
 * naw, just one big GET /?t=json
 * future new-WAPI can provide something more civilized

* cloud-backend: why does it need accounting?
 * warner: backgrounder on starter-leases, transition to leasedb-capable
   version, bootstrap after loss of leasedb
 * exarkun: could we move the leases-database out to the cloud backend,
   remove local mutable state from server
  * amazon RDS?

* direct-to-S3 mode: maybe give up accounting/GC in that case
 * would still be useful for some personal use cases
 * super hard to do strong accounting (with adversarial clients) without
   a real server
  * hacky IAM roles? eww

* leases in S3 as files with one-line-per-SI? (plus accounting
  identifier, expiration time)
 * occasional fetch, populate sqlite, run query, forget sqlite
 * sometimes delete the S3 files when they've expired
 * server writes these files on behalf of identified clients
* or leases as files in tahoe itself?
 * one account per directory, so only one writer, so no conflicts
 * written by storage server, not by clients
 * S3 holds (tahoe) SERVERNAME/leases/CLIENTID/FILES

* spectrum of options
 * 1: store lease info in non-durable efficiently-queryable location
   (not on backend): sqlite
 * 2: keep two copies, try to keep in sync
  * maybe just write whole .sqlite file into S3 after each change
 * 3: fetch and build ephemeral DB when you want to make queries, then
   throw it away
  * canonical lease table is stored as loose files in S3, occasionally
    pruned
 * 4: store info in clever loose backend way, but queries will probably
   be expensive
  * would look a lot like the old local-crawler approach, but with files
    in S3, async reads

* meejah's cloud-backend branch is still the right one to mine patches
  from (2237.cloud-backend-merge.0)

* make a new 'lafs' CLI command? with cleaner subcommand tree?
 * leave 'tahoe' as plumbing, use 'lafs' as porcelain?
 * plugins?
* adding an attenuate/diminish CLI command (to get from writecap to
  readcap)?