[tahoe-dev] git-annex and tahoe-lafs as a special remote -- in progress and testing

[Jimmy Tang]

Hi Zooko,


On Sat, May 14, 2011 at 7:18 AM, Zooko O'Whielacronx <zooko at zooko.com> wrote:
> Hi Jimmy:
>
> I looked up more details about your Tahoe-LAFS configuration for git-annex:
>
> http://git-annex.branchable.com/forum/tips:_special__95__remotes__47__hook_with_tahoe-lafs/
>
> Here are the core configurations:
>
> git config annex.tahoe-store-hook 'tahoe mkdir
> tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2 && tahoe put $ANNEX_FILE
> tahoe:$ANNEX_KEY'
> git config annex.tahoe-retrieve-hook 'tahoe get
> tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY $ANNEX_FILE'
> git config annex.tahoe-remove-hook 'tahoe rm
> tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY'
> git config annex.tahoe-checkpresent-hook 'tahoe ls
> tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY 2>&1 || echo FAIL'
>

I think when I had posted the comment I had mistyped the commands, I
was editing the .git/config file by hand to configure and test things.

> Hm, there are several things that I don't understand about this. Let's
> see if there is doc about these hooks... Oh yes, there is good doc
> here:
>
> http://git.kitenet.net/?p=git-annex;a=blob;f=doc/special_remotes/hook.mdwn;h=9a7dbf7a1909e8460e8acc5eddc83bcc99665a27;hb=e72c1c33ae60f7fe544acd0a1f1331138b880e22
>
> Okay, it appears to me that the tahoe-store-hook shown above creates a
> two-level-deep directory structure and then doesn't use it, but
> instead plonks the data into a file in the root dir (named "tahoe:")
> under a file who filename equals $ANNEX_KEY.

It's up to the user to decide that I guess, you don't have to use the
directory hashing if you don't want to, tbh when I wasn't using the
directory hashing having tahoe-lafs as a back end it felt quick, but I
haven't done any benchmarks or measurements to prove this.

>
> This is extremely inefficient, because "tahoe mkdir" is a regrettably
> slow operation (it might take around 1 second), and also because the
> tahoe: directory will grow to be quite large and all reads or writes
> of it are O(N) currently, so it will get slower and slower.
>
> Hm, also it looks to me like tahoe-retrieve-hook will fail to find the
> data that it was asked for since it is looking inside those
> directories and that's not where the file was written to. Have you
> tried it? :-)
>

heh, I did ;)  see above comment, I should be more rigorous about my
examples that I share :P

> Once it works, you should probably add Tahoe-LAFS to this list of back ends:
>
> http://git-annex.branchable.com/special_remotes/
>

WIll do, I should probably add it. It's working well enough for me at
the moment, I'm probably the only person using this setup though.

Thanks,
Jimmy


-- 
http://www.sgenomics.org/~jtang/