[volunteergrid2-l] adding files to the grid

[Billy Earney]

Zooko,

I want to thank you for the help last night on IRC,  I had to go, so I
couldn't wait for you to return..  Looking at the stats from the recent
upload and download operations, it looks like the encoding is doing about
10MBps (which is very good), but the uploading portion is only doing around
30Kbps, so it looks like the upload is the bottleneck.  My upload speed is
near (1Mbps), I checked this on speedtest.net, and it seems to be accurate.
I don't expect to get 1Mbps upload speed with tahoe-lafs since it has some
extra overhead, but I would expect at least 200KBps or so.

I'll inspect more this evening.  The stats provided on the finish links were
quite helpful! :)

-----Original Message-----
From: volunteergrid2-l-bounces at tahoe-lafs.org
[mailto:volunteergrid2-l-bounces at tahoe-lafs.org] On Behalf Of Earney, Billy
C.
Sent: Tuesday, March 01, 2011 12:58 PM
To: a small group of people sharing storage resources with Tahoe-LAFS
Subject: Re: [volunteergrid2-l] adding files to the grid

Shawn,

So let me see if I understand this correctly.  The client uploads the
encrypted file to the helper.  The helper then splits the file, and creates
the extra metadata to help put the file back together (so that the client
only needs N out of S shares).  The helper then takes the shares and
distributes them to different nodes on the grid.

Can a single client use more than 1 helper?  If so, how does that fit into
the configuration?

I would expect a helper to be beneficial when a cpu cannot keep up with the
bandwidth, but my load is very low, but it also doesn't look like my
bandwidth is saturated either..  So I'm not sure where the bottleneck is..
Any ideas of commands I can run to find the bottleneck?  If figure there are
some tahoe debug commands that would help.  I'll look into those.   Does
anyone have any other advice?

I don't mind if it takes some time to upload files, but I figure something
isn't right on my end with the level of performance I've seen..

________________________________________
From: volunteergrid2-l-bounces at tahoe-lafs.org
[volunteergrid2-l-bounces at tahoe-lafs.org] On Behalf Of Shawn Willden
[shawn at willden.org]
Sent: Tuesday, March 01, 2011 10:45 AM
To: a small group of people sharing storage resources with Tahoe-LAFS
Subject: Re: [volunteergrid2-l] adding files to the grid

On Tue, Mar 1, 2011 at 8:42 AM, Billy Earney
<billy.earney at gmail.com<mailto:billy.earney at gmail.com>> wrote:
Would it be possible to have more than 1 helper in the grid?  If so, if many
of us ran helpers that would speed up adding files to the grid, but would
the contents of the raw file be viewable by the helper?

There is no problem with having many helpers.  Only the clients that are
specifically configured to use a helper will do so, and they'll use the
helper they're pointed at (there's no automatic discovery/selection of
helpers).

The helper receives the file after AES encryption but before splitting into
shares.  So the file uploaded to the helper is the same size as the
original, but it's already encrypted.

--
Shawn
_______________________________________________
volunteergrid2-l mailing list
volunteergrid2-l at tahoe-lafs.org
http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
http://bigpig.org/twiki/bin/view/Main/WebHome