[tahoe-dev] Tahoe backend for duplicity

[Micah Anderson]

* Brian Warner <warner-tahoe at allmydata.com> [2008-11-28 16:54-0500]:
> On Thu, 27 Nov 2008 11:30:36 -0500
> Micah Anderson <micah at riseup.net> wrote:
> 
> > One thing that was brought to mind was that the traditional way of
> > doing bandwidth limiting with duplicity is to pass something like
> > '--scp-command "scp -l 256"... using the tahoe backend in duplicity I
> > imagine that specifying a scp command isn't going to do anything. Is
> > there anyway to limit the bandwidth usage?
> 
> Alas, no. http://allmydtata.org/trac/tahoe/ticket/224 is about
> bandwidth management.. we need controls for both inbound and outbound.
> It will require some support from our Foolscap network library:
> http://foolscap.lothar.com/trac/ticket/41 is about this one.

Found the same tickets when I searched myself. I also asked on #tahoe to
see what people thought, and there was an interesting discussion, if I
might summarize:

Zooko personally thinks that linux users should use operating-system
level controls for that. As would provide for functionality that can't
be done inside a single app (Tahoe) anyway. If every app has a bandwidth
limit built in, and you are running two apps, and you launch two more,
then if you want the same overall limit to take effect you have to go
back to the first two and reconfigure their limit to be half as it was.

However, zooko also pointed out that it would still be useful for tahoe
to offer some functionality, for people who are incapable of configuring
the OS or their router to do it, or for people who don't want any policy
more detailed than "make sure Tahoe doesn't use more than X GB per
minute."

Others pointed out that if you dont have root privileges you might need
this. Idnar made the astute observation that doing this kind of rate
limiting at the OS/router level is highly non-trivial.

> I've run applications under 'trickle' before, which is a ptrace-based
> userspace bandwidth limiter (I think it just stalls certain systems
> calls when it observes the traffic rate going above some level). That
> might be enough for this job.. just run your local Tahoe node under
> trickled.

I decided to give tahoe a try under trickle because my home connection
has terrible upstream (cable), I did:

trickle -v -s -u 40 tahoe start

I'm not sure if this worked because of the way tahoe processes are
spawned...

I then ran duplicity to attempt to backup my home directory:

duplicity -v 9 /home/micah tahoe:///mybackup/

This let me see what it was doing:

EXECUTE: tahoe cp /tmp/duplicity-q6C_m2-tempdir/mktemp-Q3ma71-113
mybackup:duplicity-full.2008-11-28T18:01:31-05:00.vol112.difftar.gz


The web interface seemed also to be limited, as when I loaded the stats
pages to see what was going on, it was slow. I would see these 'cp'
operations happening on the command-line, but there would be no active
operations in the web interface. This was odd, but on refresh, I might
see something. I think maybe the trickle upload limit of 40 was a little
low, because after running it all night, I only managed to send 8.78MB.

micah
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20081129/1f1943ce/attachment.pgp