[tahoe-dev] Debugging memory leaks with guppy

[Francois Deppierraz]

Hi,

On 10/22/2010 09:23 AM, Brian Warner wrote:

> My suspicion is that the ResponseCache object is living a lot longer
> than expected, and it's accumulating cached responses from lots and lots
> of generations ("versions") of the mutable file that contains a
> directory which is being modified heavily. When you say your test
> "deleted about 1000 very small files from a single directory", you're
> really making 1000 sequential changes to the same directory, right? So
> there will be 1000 mutable file writes to the same file? If my suspicion
> is right, there will be 1000 different 'verinfos' values (and N times as
> many keys in self.cache, each of which may have multiple strings in the
> set, resulting in a large number of strings left around).

The same problem could also be reproduced by repeatedly listing a single
directory. Your suspicion that the ResponseCache object is living longer
than expected sounds correct.

Each read operation on a mutable file adds a new object into the
ResponseCache even when there's already an entry for this file. This
makes me think that this ResponseCache is not actually being used.

Would it be possible that some field of the verinfo tuple (such as
offset) are different between calls to mutable.filenode._add_to_cache()
and mutable.filenode._read_from_cache()?

> Or, maybe we should get rid of ResponseCache
> altogether: I don't remember what performance improvement it tries to
> provide.

I gave a try to this approach with the following change and the memory
leak indeed disappeared!

http://github.com/ctrlaltdel/tahoe-lafs/commit/3a4dbdee1a3ea8a30559dd49f51dba40ba1661fb

François