[tahoe-dev] Fwd: incident report

[Kill Your TV]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 15 Nov 2012 11:57:42 +0000 (UTC)
"Zooko Wilcox-O'Hearn" <zooko at zooko.com> wrote:

> My guess is that the memory leak has something to do with this node
> acting as a gateway (uploading and downloading files to remote
> servers) rather than as a server. (Just because the gateway does a lot
> more complicated work than the server does.) That doesn't mean the
> memory leak is okay — I still want to fix it — but maybe you could
> help track it down by using a different process for gateway and see
> whether the gateway or the server is the one that starts using too
> much memory next time around


FWIW I think I'm seeing the same problem on occasion (= 2 times within
the last year). I suspect that it is not the server instance itself that
is the issue because when the problem is triggered it's when I'm
actively doing something, such as deep-checks on all of my aliases to
repair and add leases (or potentially stupid deletes within a for loop,
such as

    for I in $(tahoe ls URI/path); do tahoe rm URI/$I; done
).

I've never experienced the leak while the node is not being actively
used by me locally.

Just a few minutes ago, while I was in the process of deleting several
files from the clc (with my asinine for loop), in the middle of the
process I received:


socket.error: [Errno 111] Connection refused
Traceback (most recent call last):
  File "/usr/bin/tahoe", line 9, in <module>
    load_entry_point('allmydata-tahoe==1.9.2', 'console_scripts', 'tahoe')()
  File "/usr/lib/pymodules/python2.7/allmydata/scripts/runner.py", line 113, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "/usr/lib/pymodules/python2.7/allmydata/scripts/runner.py", line 99, in runner
    rc = cli.dispatch[command](so)
  File "/usr/lib/pymodules/python2.7/allmydata/scripts/cli.py", line 555, in rm
    return unlink(options, command="rm")
  File "/usr/lib/pymodules/python2.7/allmydata/scripts/cli.py", line 551, in unlink
    rc = tahoe_unlink.unlink(options, command=command)
  File "/usr/lib/pymodules/python2.7/allmydata/scripts/tahoe_unlink.py", line 32, in unlink
    resp = do_http("DELETE", url)
  File "/usr/lib/pymodules/python2.7/allmydata/scripts/common_http.py", line 60, in do_http
    c.endheaders()
  File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 780, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 761, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err


and the tahoe process was consuming ~7GB of RAM and some swap. I wasn't
able to access the WebUI so I just restarted. There's nothing
interesting in twistd.log and no incidents were reported.

I'll try looking over the docs to see if it's possible to generate an
incident report outside of the webui.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQEcBAEBAgAGBQJQp8ImAAoJEKvgwxnfCgoawNwH/iBI1lCC9n4DlLsx30ooKdJQ
2ZG5FJdM+gWr5y0OoXF6Ff3jYmoJavf2d/FvqZeNohQrecFVoKo0ZoDaCPgAB693
SrzfWBV+cz44BGuAbDtrb9EdRmT7BEf24Ikbng6W2rzu0ItGzauARpocZDQMbTpK
JjpSeXZTf0hAxLtVUTTd+Lo2n16BxZvjuj8QXyvkz+efHNSFag6iYxn+ntx5JeHX
Si3IjG7G90Ek+DDgXttS1ns6hAdEKGw5xqg7meGrt+IPXA5/0xAU3PPdmxG7suTz
X6ZAxMgLemLKctu22/8gcikTVSt3aL5uEvEcpUfgJakntV5wfW1+BTgis6+vprs=
=YJiU
-----END PGP SIGNATURE-----