Changes between Version 1 and Version 2 of Ticket #1126


Ignore:
Timestamp:
2010-07-19T02:45:48Z (14 years ago)
Author:
davidsarah
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Ticket #1126 – Description

    v1 v2  
    22
    33> [20:43] <warner> as far as overall design goes, when we get around to rewriting the uploader, I think it should have a separate code path that it follows as soon as it sees any evidence of shares already being present
     4
    45> [20:44] <warner> we want the new-upload case to work quickly, but the moment we see an alreadygot= share, we should switch into a mode where we search fairly thoroughly for all the existing shares (without allocating anything), come up with a sharemap, then replace the earlier (speculative) allocations with the ideal ones
     6
    57> [20:44] <warner> (dealing with allocation failures along the way)
     8
    69> [20:45] <warner> it's probably worth losing some of the pipelining along this path to simplify the code
     10
    711> [20:45] <zooko> neat idea
     12
    813> [20:45] <zooko> I'm not sure I agree.
     14
    915> [20:45] <zooko> Okay, afk...
     16
    1017> [20:46] <warner> of course, second-upload (when we detect existing shares) is really a form of repair, better than normal repair because we have the full plaintext available locally
     18
    1119> [20:48] <warner> so maybe what ought to happen is that we rewrite the repairer, and make the uploader's "hey there are already shares there" path do: abandon the current allocations, hand the alreadygot info and the plaintext/ciphertext filehandle to the repairer, start the repairer, wait for its results
     20
    1221> [20:50] <kpreid> second-upload is a common case if you're storing a lot of files in tahoe and locally and 'moving'/copying back and forth
     22
    1323> [20:50] <warner> hm, true
     24
    1425> [20:51] <kpreid> and given immutable directories...
     26
    1527> [20:51] <warner> we've got two sorts of frontend-duplicate detectors that might move some pressure off the uploader: the helper, and the backupdb
     28
    1629> [20:51] <kpreid> some might want to have efficient upload-the-same-thing-again, as for backups
     30
    1731> [20:51] <warner> it's kind of an open question as to where the responsibilities ought to lie
     32
    1833> [20:51] <kpreid> i.e. tahoe backup without the builtin cleverness
     34
    1935> [20:52] <warner> I think it's reasonable to add backupdb support to 'tahoe cp', and to add a contents-to-filecap table to backupdb, which would avoid network IO in the case of moving files around
     36
    2037> [20:53] <warner> (currently if you move a file, the backupdb declares a miss, and we proceed to the uploader, which will either get a Helper hit or exercise the second-upload code. In my experience, second-upload results in duplicate shares, which is a drag, so I'd prefer to avoid it)
     38
    2139> [20:55] <warner> (OTOH, having the backupdb track file-contents means that we'd do an extra hash of the file for each new upload, in addition to the subsequent CHK-computation hash. OT3H, unless the files are really large, the filesystem cache should save us from actually doing the disk IO an extra time)
     40
    2241> [20:55] <zooko> warner: 1.6.0 was changed to search for all extant shares before immutable upload, IIRC
     42
    2343> [20:56] <zooko> http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/NEWS?rev=d329759bb83ad6a0#L66
     44
    2445> [20:56] <warner> I think it sends requests to everyone, yeah (which I still consider to be a scaling problem), but I don't believe that it waits to hear from everyone before starting the upload
     46
    2547> [20:56] <warner> there's always a tradeoff between stallability and thoroughness there