[tahoe-lafs-trac-stream] [Tahoe-LAFS] #3801: Clarify behavior of overwriting in immutable uploads

[Tahoe-LAFS]

#3801: Clarify behavior of overwriting in immutable uploads
--------------------------+-----------------------------------
     Reporter:  itamarst  |      Owner:  itamarst
         Type:  task      |     Status:  new
     Priority:  normal    |  Milestone:  HTTP Storage Protocol
    Component:  unknown   |    Version:  n/a
   Resolution:            |   Keywords:
Launchpad Bug:            |
--------------------------+-----------------------------------

Comment (by itamarst):

 If I were designing this from scratch, one design would be:

 1. Upload chunks of, say, 1MB.
 2. Only allow uploads in order.
 3. Chunks are written atomically, so e.g. write of less than 1MB due to
 server crash is truncated.
 4. Conflicting writes are not allowed, which is fine given atomicity,
 chunks, and ordering.

 The use of relatively large chunks mitigates the latency issues of ordered
 writes except in cases of very high bandwidth combination very high
 latency, which is not common.

 Problems with the above approach with current code:

 1. The uploaded file combines metadata with data in a single file. Which
 means semantically you can't have same-sized chunks. Maybe? Atomicity
 seems like it'd also be harder too.
 2. Would have to add limits on overwrites, and maybe hard code the append
 only which doesn't exist at the moment.
 3. Chunk sizes are smaller due to historical reasons I don't understand.

 ---

 The current data format could be changed (it's a temporary form of
 storage, so it is easier to evolve), and given flexibility in format maybe
 the above design is unnecessary; if it's not just one big file, keeping
 out of order chunks is a lot easier e.g. in separate files or in sqlite or
 something.

 Supposedly there's a branch somewhere with sqlite implementation...

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3801#comment:2>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage