Amazon S3 compatible frontend

[francois]

Allowing existing S3 clients to access files on a Tahoe grid sounds cool.

Would it be possible to build such a S3-compatible frontend for Tahoe? Should it be implemented as a new frontend or as a proxy between the WAPI and S3 clients?

A few interesting references:

• ​S3 API documentation
• ​A Twisted API for accessing Amazon Web Services

[zooko]

Yes! Very cool! The way to do it is to make a variant of src/allmydata/storage/server.py​ which doesn't read from local disk in its _iter_share_files(), but instead reads the files from its S3 bucket (it is an S3 client and a Tahoe-LAFS storage server). Likewise variants of storage/shares.py, storage/immutable.py​, and storage/mutable.py which write their data out to S3 instead of to their local filesystem.

Probably one should first start by abstracting out just the "does this go to local disk, S3, Rackspace Cloudfiles, etc" part from all the other functionality in those four files... :-)

[davidsarah]

Replying to zooko:

Yes! Very cool! The way to do it is to make a variant of src/allmydata/storage/server.py​ which doesn't read from local disk in its _iter_share_files(), but instead reads the files from its S3 bucket (it is an S3 client and a Tahoe-LAFS storage server).

Er, that would be an S3-compatible storage backend, no? "Allowing existing S3 clients to access files on a Tahoe grid" is definitely the frontend.

Note that S3 is supposed to be a RESTful web interface (and basically is, with some quirks), so an HTTP client that doesn't make too many assumptions should be able to access either S3 servers or Tahoe webapi servers already. It may be that we can add operations to the webapi to make it more closely compatible with S3, but I doubt that we can exactly emulate S3's access control model.

[zooko]

Yes, you're right. I'm so tired I don't know my frontend from my backend.

An S3-compatible backend is interesting because of this: Redundant Array of Inexpensive Clouds: ​http://allmydata.org/~zooko/RAIC.png . :-)

An S3-compatible frontend is interesting because lots of people have tools and knowledge about how to store things with the S3 API, and they could more easil retarget those things to a Tahoe-LAFS grid.

[francois]

Replying to zooko:

Yes, you're right. I'm so tired I don't know my frontend from my backend.

An S3-compatible backend is interesting because of this: Redundant Array of Inexpensive Clouds: ​http://allmydata.org/~zooko/RAIC.png . :-)

I think that backend in already adressed by #510.

An S3-compatible frontend is interesting because lots of people have tools and knowledge about how to store things with the S3 API, and they could more easil retarget those things to a Tahoe-LAFS grid.

Yeah, that's the frontend I was talking about.

Many people are investing time in building good S3 clients nowadays. And there's probalby some of them which have already addressed many of the problems we have with 'tahoe backup', the FUSE frontends, and so on.

[davidsarah]

S3 clients will assume that files are accessible under specified names, rather than cap URIs. I'm not really sure that S3's security model is very compatible with Tahoe's, unless you have a file mapping (Access Key ID, secret key) pairs to root caps, like the SFTP and FTP frontends.