#613 new enhancement

sometimes a helper hurts instead of helping (if you need to upload less than K shares)

Reported by: zooko Owned by:
Priority: major Milestone: undecided
Component: code-network Version: 1.3.0
Keywords: upload repair helper performance bandwidth Cc: tahoe-lafs.org@…
Launchpad Bug:


If #610 were fixed so that uploader (e.g. on repair) were smart about which shares to upload, then sometimes it would decide to upload only a single share, for example if the file were encoded with M=10 and 9 shares were already being served. If it were doing that, then uploading through a helper would ironically multiply the upstream bandwidth used instead of dividing it. So in that case it might want to be clever and skip using the helper.

Change History (7)

comment:1 Changed at 2009-12-30T00:06:32Z by davidsarah

  • Keywords upload repair helper performance added

comment:2 Changed at 2009-12-30T00:09:13Z by davidsarah

  • Keywords bandwidth added

comment:3 Changed at 2010-07-21T16:07:12Z by zooko

  • Summary changed from sometimes a helper doesn't help (if you need to upload less than K shares) to sometimes a helper hurts instead of helping (if you need to upload less than K shares)

comment:4 Changed at 2015-02-09T11:25:03Z by lpirl

  • Cc tahoe-lafs.org@… added

I thought about this too. Without knowing too much about the internals, here is the idea:

Let's assume the helper knows (better) which shares to upload and which to reuse. Couldn't an uploader

  1. ask the helper for it's encoding parameters,
  2. do the encryption and the erasure coding,
  3. then send the hashes of the resulting shares
  4. and finally upload those shares, the helper reports as missing?

Additionally, if a share is unhappy, the helper could fetch it from a faster sever and re-distribute it (although it is probably a difficult decision if this is beneficial or not).

comment:5 Changed at 2015-02-10T17:45:07Z by zooko

lpirl: what would be the point of the helper if the uploader was still doing erasure coding and uploading shares?

comment:6 Changed at 2015-02-10T18:15:04Z by lpirl

Sorry for not pointing that out clearly. The idea is, that the uploader only uploads shares to the helper that are actually missing on the storage servers - or is it actually implemented this way? In other words, the uploader would ask the helper if it needs a specific share prior uploading. For shares that are available but unhappy, the helper could download and re-distribute them without requiring the client to upload them again.

comment:7 Changed at 2015-02-10T18:52:41Z by zooko

lpirl: Happiness is currently a property of files, not of shares. (See https://github.com/zooko/tahoe-lafs/blob/1382-rewrite-5/docs/upload-happiness.rst for a new document about that.)

The point of the helper is to let the uploader avoid uploading all the shares, and instead upload just a copy of the ciphertext.

If there are only 1 or a few shares that need to be uploaded, then it would probably make more sense for the uploader to upload those shares directly to the servers, and not use the helper at all.

Note: See TracTickets for help on using tickets.