[tahoe-lafs-trac-stream] [Tahoe-LAFS] #999: support multiple storage backends, including amazon s3

[Tahoe-LAFS]

#999: support multiple storage backends, including amazon s3
------------------------------+--------------------------------
     Reporter:  zooko         |      Owner:  davidsarah
         Type:  enhancement   |     Status:  closed
     Priority:  major         |  Milestone:  eventually
    Component:  code-storage  |    Version:  n/a
   Resolution:  fixed         |   Keywords:  s3-backend storage
Launchpad Bug:                |
------------------------------+--------------------------------
Changes (by amontero):

 * cc: amontero@… (added)


Old description:

> The focus of this ticket is (now) adapting the existing codebase to use
> multiple backends, rather than supporting any particular backend.
>
> We already have one backend -- the filesystem backend -- which I think
> should be a plugin in the same sense that the others will be plugins
> (i.e.: other code in tahoe-lafs can interact with a filesystem plugin
> without caring very much about how or where it is storing its files --
> otherwise it doesn't seem very extensible). If you accept this, then we'd
> need to figure out what a backend plugin should look like.
>
> There is backend-independent logic in the current server implementation
> that we wouldn't want to duplicate in every other backend implementation.
> To address this, we could start by refactoring the existing code that
> reads or writes shares on disk, to use a local backend implementation
> supporting an IStorageProvider interface (probably a fairly simplistic
> filesystem-ish API).
>
> (This involves changing the code in
> [source:src/allmydata/storage/server.py] that reads from local disk in
> its [source:src/allmydata/storage/server.py at 4164#L359
> _iter_share_files()] method, and also changing
> [source:src/allmydata/storage/shares.py at 3762 storage/shares.py],
> [source:src/allmydata/storage/immutable.py at 3871#L39
> storage/immutable.py], and
> [source:src/allmydata/storage/mutable.py at 3815#L34 storage/mutable.py]
> that write shares to local disk.)
>
> At this point all the existing tests should still pass, since we haven't
> actually changed the behaviour.
>
> Then we have to add the ability to configure new storage providers. This
> involves figuring out how to map user configuration choices to what
> actually happens when a node is started, and how the credentials needed
> to log into a particular storage backend should be specified. The
> skeletal RIStorageServer would instantiate its IStorageProvider based on
> what the user configured, and use it to write/read data, get statistics,
> and so on.
>
> Naturally, all of this would require a decent amount of documentation and
> testing, too.
>
> Once we have all of this worked out, the rest of this project (probably
> to be handled in other tickets) would be identifying what other backends
> we'd want in tahoe-lafs, then documenting, implementing, and testing
> them. We already have Amazon S3 and Rackspace as targets -- users of
> tahoe-lafs will probably have their own suggestions, and more backends
> will come up with more research.

New description:

 The focus of this ticket is (now) adapting the existing codebase to use
 multiple backends, rather than supporting any particular backend.
 We already have one backend -- the filesystem backend -- which I think
 should be a plugin in the same sense that the others will be plugins
 (i.e.: other code in tahoe-lafs can interact with a filesystem plugin
 without caring very much about how or where it is storing its files --
 otherwise it doesn't seem very extensible). If you accept this, then we'd
 need to figure out what a backend plugin should look like.
 There is backend-independent logic in the current server implementation
 that we wouldn't want to duplicate in every other backend implementation.
 To address this, we could start by refactoring the existing code that
 reads or writes shares on disk, to use a local backend implementation
 supporting an IStorageProvider interface (probably a fairly simplistic
 filesystem-ish API).
 (This involves changing the code in
 [source:src/allmydata/storage/server.py] that reads from local disk in its
 [source:src/allmydata/storage/server.py at 4164#L359 _iter_share_files()]
 method, and also changing [source:src/allmydata/storage/shares.py at 3762
 storage/shares.py], [source:src/allmydata/storage/immutable.py at 3871#L39
 storage/immutable.py], and
 [source:src/allmydata/storage/mutable.py at 3815#L34 storage/mutable.py] that
 write shares to local disk.)
 At this point all the existing tests should still pass, since we haven't
 actually changed the behaviour.
 Then we have to add the ability to configure new storage providers. This
 involves figuring out how to map user configuration choices to what
 actually happens when a node is started, and how the credentials needed to
 log into a particular storage backend should be specified. The skeletal
 RIStorageServer would instantiate its IStorageProvider based on what the
 user configured, and use it to write/read data, get statistics, and so on.
 Naturally, all of this would require a decent amount of documentation and
 testing, too.
 Once we have all of this worked out, the rest of this project (probably to
 be handled in other tickets) would be identifying what other backends we'd
 want in tahoe-lafs, then documenting, implementing, and testing them. We
 already have Amazon S3 and Rackspace as targets -- users of tahoe-lafs
 will probably have their own suggestions, and more backends will come up
 with more research.

--

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/999#comment:152>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage