[volunteergrid2-l] upload errors

[erpo41 at gmail.com]

I created a new dircap and started running

$ tahoe cp -r backup-T61p/ tahoe:

(where tahoe: is pointed at a new dircap). Performance seems good so far! :)

Question:
Is that command smart enough to avoid re-uploading the same files to the
grid each time it's run? The backup-T61p directory on my volunteergrid2
server is where my laptop creates its duplicity backups. Incremental
backups generate new files in that directory but don't change the old ones.

Thanks,
Eric


On Sat, Jun 30, 2012 at 1:04 PM, Marco Tedaldi <marco.tedaldi at gmail.com>wrote:

> Hi Eric
>
> On 30.06.2012 20:08, erpo41 at gmail.com wrote:
> > Would you go into a bit more detail regarding how you worked around the
> > problem?
> >
> I can try...
>
> > In my case, I have a single directory called backup-T61p containing 1451
> > files, the vast majority around 26MB. I can't split those files into
> > different directories.
> >
> Oh... if you can't split it into subdirectories, than my workaround
> won't help you in any way.
>
> Maybe make a script that sorts the files into subdirectoreies just for
> the backup? Or don't use the backup-command but just make a script that
> uploads the files into the grid...
>
> Or investigate and help the devs to fix the problem in the next version...
>
> > Thanks,
> > Eric
> >
> >
> > On Sat, Jun 30, 2012 at 11:46 AM, Marco Tedaldi <marco.tedaldi at gmail.com
> > <mailto:marco.tedaldi at gmail.com>> wrote:
> >
> >     Hello Eric
> >
> >     On 30.06.2012 19:22, erpo41 at gmail.com <mailto:erpo41 at gmail.com>
> wrote:
> >     > Hi all,
> >     >
> >     >
> >     >
> >     > P.S.
> >     > Here's the error I keep getting from the console:
> >     > raise
> >     >
> >
> UploadUnhappinessError(msg)\x0aallmydata.interfaces.UploadUnhappinessError:
> >     > shares could be placed on only 11 server(s) such that any 5 of
> >     them have
> >     > enough shares to recover the file, but we were asked to place
> >     shares on
> >     > at least 13 such servers. (placed all 15 shares, want to place
> >     shares on
> >     > at least 13 servers such that any 5 of them have enough shares to
> >     > recover the file, sent 11 queries to 11 servers, 11 queries placed
> >     some
> >     > shares, 0 placed none (of which 0 placed none due to the server
> being
> >     > full and 0 placed none due to an error))\x0a"
> >     >
> >     >
> >     I have had that error a lot as well when I wanted to make the whole
> >     backup in one go. surprisingly, these errors went away as soon as I
> >     started to upload the subdirectories, one by one.
> >     I did not have time to investigate is further, but there seem to be
> >     problems when trying to upload a lot of files in one process.
> >
> >     Since I know that there had been enough servers online and ready to
> >     server my request, i guess there is an error in the client. My (quite
> >     wild) guess is, that the client does not select a new set of servers
> for
> >     each file uploaded but tries to always use the same ones. And if one
> of
> >     these servers does not respond anymore, you get the happyness error.
> >
> >     As a work around you could set the total-servers higher, so it would
> >     still be happy when there are some more not accepting the data...
> >
> >     Maybe take this to tahoe-dev...
> >
> >     best regards
> >
> >     Marco
>
>
> _______________________________________________
> volunteergrid2-l mailing list
> volunteergrid2-l at tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
> http://bigpig.org/twiki/bin/view/Main/WebHome
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20120630/665a5adb/attachment-0001.html>