[tahoe-dev] using tahoe as a backend driver for irods storage system

[Jimmy Tang]

Hi Terrell,

On Mon, Dec 5, 2011 at 10:27 PM, Terrell Russell
<terrellrussell at gmail.com> wrote:
> Hi Jimmy,
>
> I'm also sitting at the intersection of these two pieces of grid-aware
> software.  I've copied the iRODS list on this reply.
>
> I suspect the performance of Tahoe-LAFS will be a major bottleneck for
> using the mass storage driver.  But that's just a hunch.  I'd love to
> see real numbers for this kind of thing instead of my idle speculation.
>  If you've already got your head around the pieces to build, give it a
> go and see if it's usable.
>

I too suspect it will be a bottleneck, I've gotten git-annex to work
with tahoe-lafs in the past with git-annex's hooks (which wouldn't be
too different to the iRODS equivalent) but performance wasn't great,
but it did do what I needed it for, I just needed a little bit more
confirmation that performance wise it would be a bit bad. I think
Zooko made some suggestions in the past to access the tahoe-lafs
system in a particular way to get good performance, I will need to dig
up some posts and comments from git-annex's wiki to find out what they
were.

> Accessing web-accessible data through iRODS microservices is another option.
>
> However, a third option, released in iRODS 3.0, is the already-included
> microservice object (MSO) drivers:
>
>
>  Microservice Object (MSO) Drivers [1]
>
>  msiobjget_http - Gets an HTTP/HTTPS/FTP object [2]
>  msiobjput_http - Puts an HTTP/HTTPS/FTP object [3]
>
>
> Please keep in touch with what you decide - iRODS currently does not
> provide encryption at the file-level, so a Tahoe-LAFS integration has
> been interesting to me for a while as well.
>

In relation to implementing a microservice I would still need to write
some rules and workflows to put data on a tahoe-grid on top of
creating a new microservice, I guess it is not the hardest thing to
do. I haven't been keeping too up to date with iRODS development
recently but I will make sure I have a look at the MSO drivers. I'm
more interested in the erase-coding capabilities of tahoe-lafs rather
than the crypto/security/protecting the data from prying eyes aspect
as I am interested in preserving data as opposed to privacy issues.

I think I need to look at the iRODS WOS driver/implementation to see
if that is a good way of approaching the problem or not. I'm both
personally and professionally interested in seeing some iRODS and
tahoe-lafs integration as well. Anyway it's just an idea for now and I
am still in the process of cooking up ideas for evaluating and
analysing if its worth doing.

Thanks,
Jimmy.


-- 
http://www.sgenomics.org/~jtang/