Changes between Version 4 and Version 5 of Ticket #999


Ignore:
Timestamp:
2010-03-31T17:17:57Z (15 years ago)
Author:
davidsarah
Comment:

Update description to reflect kevan's suggested approach.

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #999 – Description

    v4 v5  
    1 (originally I incorrectly posted this to #917)
     1The focus of this ticket is (now) adapting the existing codebase to use multiple backends, rather than supporting any particular backend.
    22
    3 The way to do it is to make a variant of [source:src/allmydata/storage/server.py] which doesn't read from local disk in its [source:src/allmydata/storage/server.py@4164#L359 _iter_share_files()], but instead reads the files using its backend protocol, e.g. from its S3 bucket (it is an S3 client and a Tahoe-LAFS storage server). Likewise variants of [source:src/allmydata/storage/shares.py@3762 storage/shares.py], [source:src/allmydata/storage/immutable.py@3871#L39 storage/immutable.py], and [source:src/allmydata/storage/mutable.py@3815#L34 storage/mutable.py] which write their data out using the backend protocol, instead of to their local filesystem.
     3We already have one backend -- the filesystem backend -- which I think should be a plugin in the same sense that the others will be plugins (i.e.: other code in tahoe-lafs can interact with a filesystem plugin without caring very much about how or where it is storing its files -- otherwise it doesn't seem very extensible). If you accept this, then we'd need to figure out what a backend plugin should look like.
    44
    5 Probably one should first start by abstracting out just the "does this go to local disk, S3, Rackspace Cloudfiles, etc" part from all the other functionality in those four files...  :-)
     5There is backend-independent logic in the current server implementation that we wouldn't want to duplicate in every other backend implementation. To address this, we could start by refactoring the existing code that reads or writes shares on disk, to use a local backend implementation supporting an IStorageProvider interface (probably a fairly simplistic filesystem-ish API).
     6
     7(This involves changing the code in [source:src/allmydata/storage/server.py] that reads from local disk in its [source:src/allmydata/storage/server.py@4164#L359 _iter_share_files()] method, and also changing [source:src/allmydata/storage/shares.py@3762 storage/shares.py], [source:src/allmydata/storage/immutable.py@3871#L39 storage/immutable.py], and [source:src/allmydata/storage/mutable.py@3815#L34 storage/mutable.py] that write shares to local disk.)
     8
     9At this point all the existing tests should still pass, since we haven't actually changed the behaviour.
     10
     11Then we have to add the ability to configure new storage providers. This involves figuring out how to map user configuration choices to what actually happens when a node is started, and how the credentials needed to log into a particular storage backend should be specified. The skeletal RIStorageServer would instantiate its IStorageProvider based on what the user configured, and use it to write/read data, get statistics, and so on.
     12
     13Naturally, all of this would require a decent amount of documentation and testing, too.
     14
     15Once we have all of this worked out, the rest of this project (probably to be handled in other tickets) would be identifying what other backends we'd want in tahoe-lafs, then documenting, implementing, and testing them. We already have Amazon S3 and Rackspace as targets -- users of tahoe-lafs will probably have their own suggestions, and more backends will come up with more research.