[tahoe-lafs-trac-stream] [tahoe-lafs] #1523: MDMF upload via web-API uses much more memory in the gateway process than expected

[tahoe-lafs]

#1523: MDMF upload via web-API uses much more memory in the gateway process than
expected
--------------------------------+------------------------------------------
     Reporter:  davidsarah      |      Owner:  davidsarah
         Type:  defect          |     Status:  assigned
     Priority:  major           |  Milestone:  undecided
    Component:  code-frontend-  |    Version:  1.9.0a1
  web                           |   Keywords:  memory tahoe-put performance
   Resolution:                  |
Launchpad Bug:                  |
--------------------------------+------------------------------------------

Comment (by davidsarah):

 For comparison, SDMF (still using 1.9alpha), has a peak gateway memory
 usage of 1314.5 MiB RSS, and then toward the end of the upload:
 {{{
   PID   RSS    VSZ S TTY          TIME COMMAND
  3284 41252 165572 S ?        00:00:00 [...]/tahoe restart
 ../grid/introducer
  3297 100980 184612 S ?       00:00:04 [...]/tahoe restart ../grid/server1
  3310 219140 302992 S ?       00:00:06 [...]/tahoe restart ../grid/server2
  3320 226224 310220 S ?       00:00:06 [...]/tahoe restart ../grid/server3
  3332 165964 249720 S ?       00:00:04 [...]/tahoe restart ../grid/server4
  3344 1085276 1213212 R ?     00:01:01 [...]/tahoe restart
  3364  4776  28488 S pts/3    00:00:00 /usr/bin/python bin/tahoe put
 --mutable --mutable-type=sdmf zeros
  3365 29840 100572 S pts/3    00:00:00 [...]/tahoe put --mutable
 --mutable-type=sdmf zeros
 }}}

 After the upload it went back down to
 {{{
   PID   RSS    VSZ S TTY          TIME COMMAND
  3284 41252 165572 S ?        00:00:00 [...]/tahoe restart
 ../grid/introducer
  3297 101092 184612 S ?       00:00:04 [...]/tahoe restart ../grid/server1
  3310 36036 119484 S ?        00:00:07 [...]/tahoe restart ../grid/server2
  3320 230492 314044 S ?       00:00:07 [...]/tahoe restart ../grid/server3
  3332 35936 119504 S ?        00:00:04 [...]/tahoe restart ../grid/server4
  3344 43628 171488 S ?        00:01:02 [...]/tahoe restart
 }}}

 So, there is no memory leak in the gateway for SDMF, although it seems
 that the storage servers are taking a while to release their memory.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1523#comment:11>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage