[tahoe-lafs-trac-stream] [tahoe-lafs] #1523: MDMF upload via web-API uses much more memory in the gateway process than updating the same file via SFTP

[tahoe-lafs]

#1523: MDMF upload via web-API uses much more memory in the gateway process than
updating the same file via SFTP
--------------------------------+------------------------------------------
     Reporter:  davidsarah      |      Owner:  davidsarah
         Type:  defect          |     Status:  assigned
     Priority:  major           |  Milestone:  undecided
    Component:  code-frontend-  |    Version:  1.9.0a1
  web                           |   Keywords:  memory tahoe-put performance
   Resolution:                  |
Launchpad Bug:                  |
--------------------------------+------------------------------------------

Comment (by davidsarah):

 Replying to [comment:2 warner]:
 > I can imagine two problems here.. which are you thinking of?
 >
 >  * SFTP only uses SDMF so far (I think), so maybe MDMF uploads use more
 memory than SDMF, regardless of how the data gets to the gateway

 SFTP will update existing mutable files that are either SDMF or MDMF. In
 this case we're using SFTP to update an MDMF file as a comparison for the
 memory that would be used if streaming were supported.

 >  * the webapi path holds temporary data in different ways than SFTP does
 (in which case we'd be comparing SFTP-to-SDMF against webapi-to-SDMF)

 No, I'm comparing SFTP-to-MDMF against webapi-to-MDMF. We expect the
 memory usage for SDMF to be bad because that would use a whole-file
 segment, and there would be more than one segment-sized buffer in memory.
 For webapi-to-MDMF, on the other hand, we should be able to have only one
 file-sized buffer in memory even without supporting streaming, and so the
 memory usage should only be worse than for SFTP-to-MDMF by approximately
 the file size.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1523#comment:3>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage