[tahoe-lafs-trac-stream] [tahoe-lafs] #1523: MDMF upload via web-API uses much more memory in the gateway process than expected (was: MDMF upload via web-API uses much more memory in the gateway process than updating the same file via SFTP)

[tahoe-lafs]

#1523: MDMF upload via web-API uses much more memory in the gateway process than
expected
--------------------------------+------------------------------------------
     Reporter:  davidsarah      |      Owner:  davidsarah
         Type:  defect          |     Status:  assigned
     Priority:  major           |  Milestone:  undecided
    Component:  code-frontend-  |    Version:  1.9.0a1
  web                           |   Keywords:  memory tahoe-put performance
   Resolution:                  |
Launchpad Bug:                  |
--------------------------------+------------------------------------------
Description changed by davidsarah:

Old description:

> Split from #113:
>
> The web-API interface does not support streaming (#113, #320), so it is
> expected for the gateway to need to hold the whole file in memory in
> order to upload it. However, when using {{{tahoe put}}} to upload an MDMF
> file, the increase in memory usage of the gateway process seems to be
> more than double the file size. For example, when uploading a 191 MiB
> MDMF file in 1.9alpha using {{{tahoe put --mutable --mutable-
> type=mdmf}}}, the peak RSS of the gateway (which was also a storage
> server in this test) was about 510 MiB greater than when updating the
> same file using SFTP (which streams to, and then from, a temporary file).

New description:

 Split from #113:

 The web-API interface does not support streaming (#113, #320), so it is
 expected for the gateway to need to hold the whole file in memory in order
 to upload it. However, when using {{{tahoe put}}} to upload an MDMF file,
 the increase in memory usage of the gateway process seems to be more than
 double the file size. For example, when uploading a 191 MiB MDMF file in
 1.9alpha using {{{tahoe put --mutable --mutable-type=mdmf}}}, the peak RSS
 of the gateway (which was also a storage server in this test) was over
 1300 MiB. There is also a huge memory leak of more than 700 MiB after the
 upload has finished.

 I originally thought that the memory usage was larger when using the web-
 API than when updating the same file using SFTP, but apparently that was
 wrong (I may have been misled by at first doing the SFTP experiment
 without restarting the nodes).

--

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1523#comment:7>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage