[tahoe-dev] [tahoe-lafs] #1200: package up Brian's New Visualization of immutable download

[tahoe-lafs]

#1200: package up Brian's New Visualization of immutable download
-----------------------------+----------------------------------------------
     Reporter:  zooko        |       Owner:  nobody                                                                    
         Type:  enhancement  |      Status:  new                                                                       
     Priority:  major        |   Milestone:  1.9.0                                                                     
    Component:  unknown      |     Version:  1.8β                                                                      
   Resolution:               |    Keywords:  unfinished-business immutable download statistics performance transparency
Launchpad Bug:               |  
-----------------------------+----------------------------------------------

Comment (by warner):

 The viz tool requires two Javascript libraries: jQuery (121KB source,
 57KB minified) and Protovis (510KB source, 117KB minified). It also
 contains a small JS program that is downloaded as part of the
 "download-status" WUI page, included in {{{src/allmydata/web/}}} .

 We identified a couple of goals/concerns around packaging this JS code:

 1. we'd prefer to not include non-Tahoe sources in the Tahoe VC tree.
 2. we only want to manage source code in our VC tree, not
    minified/compressed JS (which is the moral equivalent of object code)
 3. we don't want to store large things in the Tahoe VC tree, compared to
    the 3MB of .py files in src/
 4. we don't want Tahoe user security to depend upon anything beyond
    their own Tahoe node. This rules out having the viz page load the
    libraries from their home pages or from google.

 We used to have zfec and Crypto in src/, and removed them. We managed to
 remove the random windows .exe files from the tree. It'd be nice to
 avoid adding such things back in.

 Unfortunately, there are few established practices for packaging
 Javascript files on the desktop (most web sites either serve their own
 copy or rely upon Google's fast servers). Our current tool for
 obtaining/building dependencies (setuptools) only handles python, not
 JS.

 Debian does appear to have the beginnings of a [http://ianlawrence.info
 /random-stuff/packaging-web-applications-in-debian JS packaging policy]
 and has a [http://packages.debian.org/lenny/libjs-jquery libjs-jquery]
 package (which, when installed together with the
 [http://packages.debian.org/lenny/javascript-common javascript-common]
 package, makes {{{http://HOSTNAME/javascript/jquery/jquery.min.js}}}
 available). This makes it feasible for web apps on a debian box to all
 share the same (upgradable) copy of common libraries.

 We seem to have consensus on the importance of goal 4: Tahoe is very
 focussed on not relying upon outside parties for security, and until
 browsers offer a way to process and enforce cryptographic hashes in URLs
 (so we could specify exactly what script we wanted to pull from e.g.
 google), we're not comfortable with extending the TCB to include
 external web sites, much less the "anyone-who-can-spoof-your-DNS" attack
 enabled by using http URLs (instead of https).

 Some potential solutions we've discussed:

 A. bite-the-bullet: commit copies of the JS libraries into Tahoe's VC
    tree, probably the non-minified form to retain goal 2
 B. build-time: add a build step which downloads the libraries from some
    well-known place, checks their hashes against precomputed
    expectations, and copies them into the source tree somewhere
    ({{{support/lib/js}}}?)
  * this covers run-from-source, but packagers will have to come up with
    something else. Debian packages could depend upon libjs-jquery and use
    symlinks to reach the files, but there is no protovis package yet.
 Other
    OSes are unlikely to have even that.
 C. plugin: define a plugin interface for Tahoe, and put all the viz
    pieces in a new project. The new project's VC repo would have
    precisely the same problem, but maybe we'd be willing to
    bite-the-bullet more readily on the plugin than on the main Tahoe
    repo.
  * For the viz plugin specifically, we'd need two hooks:
   * add a link to each download-status page
   * add code to serve a new page (the viz JS program) from that link
   * (the basic JSON data source could be built-in to the Tahoe core, as
 it's
     useful to more than just a JS-based viz tool)
 D. separate process: if we enable CORS on the JSON data source, the viz
    program could conceivably be served from a separate program (on a
    separate port), but it would take some fussing to let it find out
    what download it should display (the program would need to scrape the
    recent-uploads-and-downloads page to get the download numbers, then
    offer its own menu page)
 E. webapp-in-grid: store the JS viz program in the same grid that the
    Tahoe node is using, then let it behave like the separate-process
    case.

 B (build-time) would be disappointing, because I'm trying to get rid of
 the
 "build" step (#479). My
 [https://github.com/warner/tahoe-lafs/tree/unsuck "unsuck"] branch splits
 the
 build step into two pieces: one is a helper which checks to see if you
 have
 the dependencies available, and the other will download+build them for
 you:
 the idea is that there may be more appropriate ways to obtain the
 dependencies (i.e. {{{apt-get install python-zfec}}}), so download+build
 shouldn't be the default. When OS packages of the JS libraries are
 available,
 this could be done cleanly, but otherwise we have to define what it means
 for
 each JS library to be "available" (i.e. where it lives).

 C (plugin) would be cool for other reasons, but is a lot of work and
 yak-shaving for a fairly small feature. It also just pushes the problem
 elsewhere.

 D (separate process) doesn't sound likely to provide a good user
 experience.
 Download status charts should be reached from a link on the download-
 status
 summary page, not from a completely separate app.

 E (webapp-in-grid) has the same problem, plus I'm uncomfortable with the
 idea
 of storing pieces of Tahoe's functionality in Tahoe itself, both because
 of
 the setup problem (what, should each copy of tahoe auto-upload the JS app
 into each grid it touches upon first boot?), and because of the recursion
 problem (what tool do you use to diagnose download problems of the
 download-status JS?).


 So I'm undecided. I guess I lean towards B, as it seems to be the cleanest
 option. I wish the libraries were small enough that we could just jam them
 into the tree (option A), but they're not. In the long run, C (plugins)
 would
 make it easier to get new functionality into the node, and I bet we'd be
 willing to commit jQuery into the plugin's repo (especially since then, if
 we
 changed our mind, we could just ditch that repo and start another plugin),
 but it doesn't feel right for the near-term.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1200#comment:8>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage