maybe have separate code paths for upload and repair

[davidsarah]

IRC discussion:

[20:43] <warner> as far as overall design goes, when we get around to rewriting the uploader, I think it should have a separate code path that it follows as soon as it sees any evidence of shares already being present

[20:44] <warner> we want the new-upload case to work quickly, but the moment we see an alreadygot= share, we should switch into a mode where we search fairly thoroughly for all the existing shares (without allocating anything), come up with a sharemap, then replace the earlier (speculative) allocations with the ideal ones

[20:44] <warner> (dealing with allocation failures along the way)

[20:45] <warner> it's probably worth losing some of the pipelining along this path to simplify the code

[20:45] <zooko> neat idea

[20:45] <zooko> I'm not sure I agree.

[20:45] <zooko> Okay, afk...

[20:46] <warner> of course, second-upload (when we detect existing shares) is really a form of repair, better than normal repair because we have the full plaintext available locally

[20:48] <warner> so maybe what ought to happen is that we rewrite the repairer, and make the uploader's "hey there are already shares there" path do: abandon the current allocations, hand the alreadygot info and the plaintext/ciphertext filehandle to the repairer, start the repairer, wait for its results

[20:50] <kpreid> second-upload is a common case if you're storing a lot of files in tahoe and locally and 'moving'/copying back and forth

[20:50] <warner> hm, true

[20:51] <kpreid> and given immutable directories...

[20:51] <warner> we've got two sorts of frontend-duplicate detectors that might move some pressure off the uploader: the helper, and the backupdb

[20:51] <kpreid> some might want to have efficient upload-the-same-thing-again, as for backups

[20:51] <warner> it's kind of an open question as to where the responsibilities ought to lie

[20:51] <kpreid> i.e. tahoe backup without the builtin cleverness

[20:52] <warner> I think it's reasonable to add backupdb support to 'tahoe cp', and to add a contents-to-filecap table to backupdb, which would avoid network IO in the case of moving files around

[20:53] <warner> (currently if you move a file, the backupdb declares a miss, and we proceed to the uploader, which will either get a Helper hit or exercise the second-upload code. In my experience, second-upload results in duplicate shares, which is a drag, so I'd prefer to avoid it)

[20:55] <warner> (OTOH, having the backupdb track file-contents means that we'd do an extra hash of the file for each new upload, in addition to the subsequent CHK-computation hash. OT3H, unless the files are really large, the filesystem cache should save us from actually doing the disk IO an extra time)

[20:55] <zooko> warner: 1.6.0 was changed to search for all extant shares before immutable upload, IIRC

[20:56] <zooko> ​http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/NEWS?rev=d329759bb83ad6a0#L66

[20:56] <warner> I think it sends requests to everyone, yeah (which I still consider to be a scaling problem), but I don't believe that it waits to hear from everyone before starting the upload

[20:56] <warner> there's always a tradeoff between stallability and thoroughness there