[volunteergrid2-l] Fwd: [volunteergrid-l] Grid nonfunctional for me

Tue Aug 30 07:46:12 PDT 2011

Hi Billy,

I don't know if some VG1 users might be interested in coming over.  I'm
waiting to see if anyone expresses great frustration, and then I'll pounce
:-)

BTW, I'm sorry I haven't responded to your e-mail about GridBackup.  I'm
trying not to let myself get distracted with other stuff until I get the
house unpacked.  Still, I have a few minutes right now, so I'll try to
answer your questions here.

First, the general status of GridBackup is... mixed.  The pure Python
implementation works fairly well, but only has backup. Restore isn't
implemented.  But I've actually mostly abandoned that code in favor of a
rewrite that is even less functional.  The reason for the rewrite is because
I decided I needed to fundamentally change the architecture to a
client-server model.  Most stuff that I need to back up these days lives on
laptops which have only intermittent network connections (and are sleeping
or off a lot), and aren't good candidates for being Tahoe nodes, or for
running GridBackup in its original form.

So, I started a rewrite that is more of a client/server architecture.  A
GridBackup server generally lives right next to a Tahoe node and uses it to
push stuff to the grid.  A GridBackup client lives on the machine needing
backup services, and it streams data at full local network speed to a
GridBackup server, which takes ownership of the data and then starts pushing
it out to the grid, at whatever speed can be managed (generally much slower
than LAN speeds).

I also decided to write the client in C++ rather than Python, because Python
was difficult to get working on Windows, and because I decided that the
clients ultimately need to look and feel like "native" apps on the various
platforms.  And also because I enjoy writing C++ :-)  (I am willing to
revisit that decision -- and one nice thing about a client/server
architecture is that multiple client implementations are not only possible,
but a good thing).

So, I think that addresses your questions from the other e-mail.

As for this one... I had to go back and read the code, and it looks like
this is a bug.  For packed files, while only one pack file gets uploaded, so
there's only one read cap, there are many read cap index entries, one for
each file in the pack (I'm sure you know this, I'm explaining partly for
myself and partly for anyone else who might care to read this).  Each read
cap index entry for a file in a pack should contain the offset of the file
within the pack (the fourth field in the index structure) and the length of
the file (the ninth and last field in the index structure).

However, the code obtains the file size not by having it passed in through
the file info structure but by extracting it from the read cap.  But there's
only one read cap per pack file, and the size it contains is the size of the
whole pack file, not the individual file.  Doh!  This is why it's bad to
write the backup side without also writing the restore side at the same
time, so you can create unit tests that ensure that what you get back is
what you put in!

I'll take a look at it and fix it in a few days... or you're more than
welcome to fix it and send me a pull request :-)

On Tue, Aug 30, 2011 at 7:30 AM, Billy Earney <billy.earney at gmail.com>wrote:

> Shawn,
>
> Would this user be interested in joining our grid?
>
> By the way, I'm looking at your GridBackup code.  That seems like a very
> interesting and useful project.  I'm looking into creating a command line
> recover script, and I notice that for the files that get packed together in
> a single file (for upload) all have the same data_size, even though many of
> the files I'm backing up are different sizes.  The offset values are
> different and so far seem accurate.  Have you run into this type of problem?
>
> Thanks!
>
> On Mon, Aug 29, 2011 at 9:45 AM, Shawn Willden <shawn at willden.org> wrote:
>
>> FYI, _this_ is why we have uptime and storage requirements.  Many VG1
>> nodes are down, and some of the nodes that are up are full; the result is
>> that the grid is unavailable for writes unless you have a low shares.happy
>> value.
>>
>> This situation is one that occurs every few months on VG1.  Let's make
>> sure it doesn't happen on VG2!
>>
>> ---------- Forwarded message ----------
>> Date: Mon, Aug 29, 2011 at 7:44 AM
>> Subject: [volunteergrid-l] Grid nonfunctional for me
>> To: volunteergrid-l at allmydata.org
>>
>>
>> I have not been able to upload small files since at least two days ago and
>> possibly much longer. The grid is being essentially useless for my need. My
>> current view of the grid is attached.
>>
>> Is anyone else seeing more servers than I am?
>>
>> Are any servers specifically offline due to the current storm?
>>
>> Can anyone bring up more storage?
>>
>> Everyone got lease expiry running?
>>
>> Service Name
>> Nickname
>> PeerID
>> Connected?Since First AnnouncedVersion storage
>> ndurner at imp-ap01
>> asrcmetfrh3k5ecusxasosibe7djzaza
>> No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/1.4.1storage
>> slush at future
>> bwxr6cbkutq2s5slg7dhh5ukejn6567i
>> No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/1.8.0storage
>> lilith.goop.org
>> gtnnoahwi5ukb4ztjs3un7mh7joopb33
>> No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/unknownstorage
>> slush at beagle
>> hpdlcilo7cjag7c4ffzehavh2wdv2zrp
>> No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/1.8.2storage
>> shawn at willden.org-gratch
>> iutxbkwuy5ekowafh7r4nrxl3v446irx
>> No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/1.8.1storage
>> kpreid at gobi
>> m2sbabta5mc22ijtfkcidmpwbuxbp3bf
>> Yes: to 67.249.228.147:5103919:10:21 28-Aug-2011 19:10:20 28-Aug-2011
>> allmydata-tahoe/1.5.0-r4126 storage
>> Kamina
>> n22vycllmqvq23plyiuxyqmzdfwpk44l
>> No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/1.8.2storage
>> Vapor
>> sarpx7ziac3wd6lbbtnujsjhym3mjhgi
>>  No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/1.8.1-r4912storage
>> francois1 at tahoe.ctrlaltdel.ch
>> xlnzwmko5inqjs4krxk35yj2sqjtb6ma
>> Yes: to 62.220.136.29:5002919:10:21 28-Aug-2011 19:10:20 28-Aug-2011
>> allmydata-tahoe/1.4.1-r3916 storage
>> kpreid at eider
>> xpu6hdv46lnkzfxox5tfn36mo3oo25eh
>> Yes: to (loopback)19:10:20 28-Aug-2011 19:10:20 28-Aug-2011
>> allmydata-tahoe/1.8.2-r5062 storage
>> FreeStorm-Alix2
>> x2l47knnwwvfxonturznfh3n5hh7g7a2
>> Yes: to 178.198.162.108:808919:10:22 28-Aug-2011 19:10:20 28-Aug-2011
>> allmydata-tahoe/1.8.1 storage
>> in.struc.tv
>> ysxpoywqf37dpaxxuonoqc3bod24ic6w
>> Yes: to 173.255.230.135:4429019:10:21 28-Aug-2011 19:10:20 28-Aug-2011
>> allmydata-tahoe/1.8.0 storage
>> FreeStorm-Alix4
>> 3cragbava4skmnrx46sk2kegly7wrjp4
>> No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/1.8.1storage
>> trelbox
>> 5ouv6kibslfodqbmifie3kfru2v5jnd7
>>  No09:34:57 29-Aug-201119:10:20 28-Aug-2011 allmydata-tahoe/1.8.2-r5048storage
>> dune3
>> 63hhqm64talajpn5zu343ui3bhi4tfcc
>> Yes: to 18.4.249.35:4803219:10:21 28-Aug-2011 19:10:20 28-Aug-2011
>> allmydata-tahoe/1.7.0
>>
>> --
>> Shawn
>>
>> _______________________________________________
>> volunteergrid2-l mailing list
>> volunteergrid2-l at tahoe-lafs.org
>> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>
>
>
> _______________________________________________
> volunteergrid2-l mailing list
> volunteergrid2-l at tahoe-lafs.org
> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
> http://bigpig.org/twiki/bin/view/Main/WebHome
>

-- 
Shawn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20110830/cdc874cf/attachment-0001.html>