[tahoe-lafs-trac-stream] [tahoe-lafs] #1513: memory usage in MDMF publish

[tahoe-lafs]

#1513: memory usage in MDMF publish
-------------------------+-------------------------------------------------
     Reporter:  warner   |      Owner:
         Type:  defect   |     Status:  new
     Priority:  major    |  Milestone:  eventually
    Component:  code-    |    Version:  1.9.0a1
  mutable                |   Keywords:  mutable mdmf memory-leak
   Resolution:           |  performance docs
Launchpad Bug:           |
-------------------------+-------------------------------------------------
Changes (by zooko):

 * milestone:  1.11.0 => eventually


Old description:

> I did a 'tahoe push --mdmf --mutable-type=mdmf foo' of a 210MB file. The
> client process swelled to 1.15GB RSS, making my entire system pretty
> unresponsive. The publish eventually succeeded, and the memory usage went
> back to normal.
>
> I'm guessing that either there's a design problem in which it's trying to
> upload all segments in parallel, or there's a failure in the Pipeline
> code such that it's holding all shares in memory at the same time.
>
> Since MDMF is supposed to make it possible to work with large files, I
> think the memory usage should be similar to CHK files: capped at a small
> constant times the segsize.
>
> It would be nice to fix this for 1.9, but since MDMF is still
> experimental, I'm willing to ship without it.

New description:

 I did a 'tahoe push --mdmf --mutable-type=mdmf foo' of a 210MB file. The
 client process swelled to 1.15GB RSS, making my entire system pretty
 unresponsive. The publish eventually succeeded, and the memory usage went
 back to normal.

 I'm guessing that either there's a design problem in which it's trying to
 upload all segments in parallel, or there's a failure in the Pipeline code
 such that it's holding all shares in memory at the same time.

 Since MDMF is supposed to make it possible to work with large files, I
 think the memory usage should be similar to CHK files: capped at a small
 constant times the segsize.

 It would be nice to fix this for 1.9, but since MDMF is still
 experimental, I'm willing to ship without it.

--

Comment:

 This isn't going to make it into 1.10.0. I think it requires a deep
 change. Ultimately I think it actually requires end-to-end two-phase-
 commit (#1755)!

 Let's see, does the docs/performance.rst already document this issue?
 [source:trunk/docs/performance.rst?rev=514fb096be50464ce78933f4db48db4de40e7265
 #publishing-an-a-byte-mutable-file]. Yes! Good.

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1513#comment:8>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage