Changes between Initial Version and Version 1 of Ticket #1126


Ignore:
Timestamp:
2010-07-19T02:44:31Z (11 years ago)
Author:
davidsarah
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Ticket #1126 – Description

    initial v1  
    1 {{{
    2 [20:43] <warner> as far as overall design goes, when we get around to rewriting the uploader, I think it should have a separate code path that it follows as soon as it sees any evidence of shares already being present
    3 [20:44] <warner> we want the new-upload case to work quickly, but the moment we see an alreadygot= share, we should switch into a mode where we search fairly thoroughly for all the existing shares (without allocating anything), come up with a sharemap, then replace the earlier (speculative) allocations with the ideal ones
    4 [20:44] <warner> (dealing with allocation failures along the way)
    5 [20:45] <warner> it's probably worth losing some of the pipelining along this path to simplify the code
    6 [20:45] <zooko> neat idea
    7 [20:45] <zooko> I'm not sure I agree.
    8 [20:45] <zooko> Okay, afk...
    9 [20:46] <warner> of course, second-upload (when we detect existing shares) is really a form of repair, better than normal repair because we have the full plaintext available locally
    10 [20:48] <warner> so maybe what ought to happen is that we rewrite the repairer, and make the uploader's "hey there are already shares there" path do: abandon the current allocations, hand the alreadygot info and the plaintext/ciphertext filehandle to the repairer, start the repairer, wait for its results
    11 [20:50] <kpreid> second-upload is a common case if you're storing a lot of files in tahoe and locally and 'moving'/copying back and forth
    12 [20:50] <warner> hm, true
    13 [20:51] <kpreid> and given immutable directories...
    14 [20:51] <warner> we've got two sorts of frontend-duplicate detectors that might move some pressure off the uploader: the helper, and the backupdb
    15 [20:51] <kpreid> some might want to have efficient upload-the-same-thing-again, as for backups
    16 [20:51] <warner> it's kind of an open question as to where the responsibilities ought to lie
    17 [20:51] <kpreid> i.e. tahoe backup without the builtin cleverness
    18 [20:52] <warner> I think it's reasonable to add backupdb support to 'tahoe cp', and to add a contents-to-filecap table to backupdb, which would avoid network IO in the case of moving files around
    19 [20:53] <warner> (currently if you move a file, the backupdb declares a miss, and we proceed to the uploader, which will either get a Helper hit or exercise the second-upload code. In my experience, second-upload results in duplicate shares, which is a drag, so I'd prefer to avoid it)
    20 [20:55] <warner> (OTOH, having the backupdb track file-contents means that we'd do an extra hash of the file for each new upload, in addition to the subsequent CHK-computation hash. OT3H, unless the files are really large, the filesystem cache should save us from actually doing the disk IO an extra time)
    21 [20:55] <zooko> warner: 1.6.0 was changed to search for all extant shares before immutable upload, IIRC
    22 [20:56] <zooko> http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/NEWS?rev=d329759bb83ad6a0#L66
    23 [20:56] <warner> I think it sends requests to everyone, yeah (which I still consider to be a scaling problem), but I don't believe that it waits to hear from everyone before starting the upload
    24 [20:56] <warner> there's always a tradeoff between stallability and thoroughness there
    25 }}}
     1IRC discussion:
     2
     3> [20:43] <warner> as far as overall design goes, when we get around to rewriting the uploader, I think it should have a separate code path that it follows as soon as it sees any evidence of shares already being present
     4> [20:44] <warner> we want the new-upload case to work quickly, but the moment we see an alreadygot= share, we should switch into a mode where we search fairly thoroughly for all the existing shares (without allocating anything), come up with a sharemap, then replace the earlier (speculative) allocations with the ideal ones
     5> [20:44] <warner> (dealing with allocation failures along the way)
     6> [20:45] <warner> it's probably worth losing some of the pipelining along this path to simplify the code
     7> [20:45] <zooko> neat idea
     8> [20:45] <zooko> I'm not sure I agree.
     9> [20:45] <zooko> Okay, afk...
     10> [20:46] <warner> of course, second-upload (when we detect existing shares) is really a form of repair, better than normal repair because we have the full plaintext available locally
     11> [20:48] <warner> so maybe what ought to happen is that we rewrite the repairer, and make the uploader's "hey there are already shares there" path do: abandon the current allocations, hand the alreadygot info and the plaintext/ciphertext filehandle to the repairer, start the repairer, wait for its results
     12> [20:50] <kpreid> second-upload is a common case if you're storing a lot of files in tahoe and locally and 'moving'/copying back and forth
     13> [20:50] <warner> hm, true
     14> [20:51] <kpreid> and given immutable directories...
     15> [20:51] <warner> we've got two sorts of frontend-duplicate detectors that might move some pressure off the uploader: the helper, and the backupdb
     16> [20:51] <kpreid> some might want to have efficient upload-the-same-thing-again, as for backups
     17> [20:51] <warner> it's kind of an open question as to where the responsibilities ought to lie
     18> [20:51] <kpreid> i.e. tahoe backup without the builtin cleverness
     19> [20:52] <warner> I think it's reasonable to add backupdb support to 'tahoe cp', and to add a contents-to-filecap table to backupdb, which would avoid network IO in the case of moving files around
     20> [20:53] <warner> (currently if you move a file, the backupdb declares a miss, and we proceed to the uploader, which will either get a Helper hit or exercise the second-upload code. In my experience, second-upload results in duplicate shares, which is a drag, so I'd prefer to avoid it)
     21> [20:55] <warner> (OTOH, having the backupdb track file-contents means that we'd do an extra hash of the file for each new upload, in addition to the subsequent CHK-computation hash. OT3H, unless the files are really large, the filesystem cache should save us from actually doing the disk IO an extra time)
     22> [20:55] <zooko> warner: 1.6.0 was changed to search for all extant shares before immutable upload, IIRC
     23> [20:56] <zooko> http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/NEWS?rev=d329759bb83ad6a0#L66
     24> [20:56] <warner> I think it sends requests to everyone, yeah (which I still consider to be a scaling problem), but I don't believe that it waits to hear from everyone before starting the upload
     25> [20:56] <warner> there's always a tradeoff between stallability and thoroughness there