[tahoe-lafs-trac-stream] [tahoe-lafs] #1395: error doing a check --verify on files bigger than about 1Gbyte

[tahoe-lafs]

#1395: error doing a check --verify on files bigger than about 1Gbyte
-------------------------------+---------------------------------
     Reporter:  sickness       |      Owner:  nobody
         Type:  defect         |     Status:  new
     Priority:  minor          |  Milestone:  undecided
    Component:  code-encoding  |    Version:  1.8.2
   Resolution:                 |   Keywords:  memory verify error
Launchpad Bug:                 |
-------------------------------+---------------------------------

Comment (by warner):

 Not that it's likely to be the problem here, but we should update that
 claim to be "{{{N/K*S}}} times a small multiple". I think the multiple is
 currently about 2 or 3. During encryption, we hold both a plaintext share
 and a ciphertext share in RAM at the same time (so 2*S), then we drop the
 plaintext. During erasure-coding, we hold a whole S of ciphertext in
 memory at the same time as the N/K*S shares, then we drop the ciphertext
 before pushing. We also pipeline the sends a little bit, I think 10kB or
 50kB per server, to get better utilization out of a non-zero-latency wire.

 Also Python's memory-management strategy interacts weirdly. Dropping the
 plaintext segment may not be enough: Python might not re-use that memory
 space for anything else right away. Although I'd expect it to de-fragment
 or coalesce free blocks before asking the OS for so much memory that it
 crashed.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1395#comment:12>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage