[tahoe-lafs-trac-stream] [tahoe-lafs] #1523: MDMF upload via web-API uses much more memory in the gateway process than updating the same file via SFTP

[tahoe-lafs]

#1523: MDMF upload via web-API uses much more memory in the gateway process than
updating the same file via SFTP
--------------------------------+------------------------------------------
     Reporter:  davidsarah      |      Owner:  davidsarah
         Type:  defect          |     Status:  assigned
     Priority:  major           |  Milestone:  undecided
    Component:  code-frontend-  |    Version:  1.9.0a1
  web                           |   Keywords:  memory tahoe-put performance
   Resolution:                  |
Launchpad Bug:                  |
--------------------------------+------------------------------------------

Comment (by davidsarah):

 I restarted all the nodes again, and logged in using {{{sftp -P 8022
 127.0.0.1}}}. The memory usage at that point was:
 {{{
   PID   RSS    VSZ S TTY          TIME COMMAND
 29546 39984 163876 S ?        00:00:00 [...]/tahoe start
 ../grid/introducer
 29566 35408 118816 S ?        00:00:00 [...]/tahoe start ../grid/server1
 29593 35736 119120 S ?        00:00:00 [...]/tahoe start ../grid/server2
 29604 35552 119144 S ?        00:00:00 [...]/tahoe start ../grid/server3
 29614 35388 119048 S ?        00:00:00 [...]/tahoe start ../grid/server4
 29624 44040 173988 S ?        00:00:00 [...]/tahoe start
 }}}

 I issued the SFTP client command {{{put zeros /uri/<URI from first
 run>}}}, i.e. updating the same MDMF file.

 The results were almost identical to the first experiment (except that the
 gateway memory usage only started after the file had been completely
 received over SFTP). The peak RSS of the gateway was 1386 MiB, and the
 final usages were:
 {{{
   PID   RSS    VSZ S TTY          TIME COMMAND
 29546 38480 163876 S ?        00:00:00 [...]/tahoe start
 ../grid/introducer
 29566 180072 266984 S ?       00:00:05 [...]/tahoe start ../grid/server1
 29593 134600 221044 S ?       00:00:06 [...]/tahoe start ../grid/server2
 29604 203136 288520 S ?       00:00:08 [...]/tahoe start ../grid/server3
 29614 286520 373392 S ?       00:00:08 [...]/tahoe start ../grid/server4
 29624 763504 896112 S ?       00:01:42 [...]/tahoe start
 }}}
 (i.e. 745.6 MiB RSS for the gateway). These also hadn't decreased several
 minutes later, or on closing the SFTP connection.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1523#comment:6>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage