memory leak on 'tahoe get'

[T_X]

When trying to retrieve a file either via sshfs or via 'tahoe get' I seem to have a memory leak. Every 'tahoe get' on a 350MB large file seems to grow the memory usage of the tahoe daemon by about 20MB:

$ for i in `seq 1 5`; do ps -o %mem,cmd 2303; tahoe get "root:path/to/myfile" /dev/null; done; ps -o %mem,cmd 2303
%MEM CMD
23.3 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
26.8 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
30.9 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
34.7 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
38.8 /usr/bin/python /usr/bin/tahoe start
root:path/to/myfile retrieved and written to /dev/null
%MEM CMD
39.1 /usr/bin/python /usr/bin/tahoe start
$ free -m
             total       used       free     shared    buffers     cached
Mem:           497        267        229          0          7         37
-/+ buffers/cache:        222        274
Swap:          258          0        258

'myfile' is an immutable file according to the web interface. sshfs is unmounted for tahoe, so the 'tahoe get's above should be the only tahoe commands invoked during that time.

Versions (as provided by Debian wheezy):

allmydata-tahoe: 1.9.2, foolscap: 0.6.4, pycryptopp: 0.5.29, zfec: 1.4.5, Twisted: 12.0.0, Nevow: 0.10.0, zope.interface: unknown, python: 2.7.3rc2, platform: Linux-debian_wheezy/sid-x86_64-64bit_ELF, pyOpenSSL: 0.13, simplejson: 2.5.2, pycrypto: 2.6, pyasn1: unknown, mock: 0.8.0, sqlite3: 2.6.0 [sqlite 3.7.13], setuptools: 0.6 [distribute] 

Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.32-1 x86_64 unknown unknown GNU/Linux

Encoding parameters: shares.needed = 1, shares.happy = 2.

[davidsarah]

Does this pattern continue for more than, say, 200 MB without freeing any memory? Garbage collection doesn't necessarily happen immediately, so we need to exclude the possibility that it's just "floating garbage" rather than a leak.

[zooko]

T_X: could you please do this experiment? Keep iterating the tahoe get's and see if the memory usage keeps rising indefinitely or if it levels off. I'm guessing it is the latter, which means that this isn't a "memory leak" per se, but it might still be a problem if the level at which it levels off is too high. Please report back! Also, what sort of machine is that? Consider pasting in the output (or part of the output) of python misc/build_helpers/show-tool-versions.py.

Re-assigning this ticket to T_X.

[T_X]

Ok, I temporarily increased the RAM from 256MB to 1.5GB in this VM instance and redid the tests with 50 'tahoe get's. You are right, pretty much exactly after 10 rounds (~200MB) the memory consumption does not increase as quickly as before anymore.

However it looks like it still, linearly increases by about 0.6MB per 'tahoe get', with one 'tahoe get' needing 2 minutes. At least during these 50 rounds needing 92 minutes in total - not sure whether this curve will stop increasing after the next n rounds.

I also verified that these 0.6MB per 2 minutes are due to the 'tahoe get' invocation: Four hours after this test run the the memory consumption still is at 18.1%.

Also see attached files for details.

[davidsarah]

The section of the graph after 11 invocations does indeed appear to show a 0.6 MB-per-invocation memory leak. ps -o %mem is based on virtual size, though, and we prefer to measure RSS. Can you repeat the test using ps -o rss?