[tahoe-lafs-trac-stream] [tahoe-lafs] #1523: MDMF upload via web-API uses much more memory in the gateway process than updating the same file via SFTP

[tahoe-lafs]

#1523: MDMF upload via web-API uses much more memory in the gateway process than
updating the same file via SFTP
-------------------------------+---------------------------
 Reporter:  davidsarah         |          Owner:
     Type:  defect             |         Status:  new
 Priority:  major              |      Milestone:  undecided
Component:  code-frontend-web  |        Version:  1.9.0a1
 Keywords:  memory tahoe-put   |  Launchpad Bug:
-------------------------------+---------------------------
 Split from #113:

 The web-API interface does not support streaming (#113, #320), so it is
 expected for the gateway to need to hold the whole file in memory in order
 to upload it. However, when using {{{tahoe put}}} to upload an MDMF file,
 the increase in memory usage of the gateway process seems to be more than
 double the file size. For example, when uploading a 191 MiB MDMF file in
 1.9alpha using {{{tahoe put --mutable --mutable-type=mdmf}}}, the peak RSS
 of the gateway (which was also a storage server in this test) was about
 510 MiB greater than when updating the same file using SFTP (which streams
 to, and then from, a temporary file).

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1523>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage