webapi fails to handle all TCP disconnects: "Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this."

[nejucomo]

Symptoms: While running a gateway which has haproxy as a webapi client which I frequently restart, I see exceptions like this:

2012-01-21 14:35:15-0800 [-] Unhandled error in Deferred:
2012-01-21 14:35:15-0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/flat/twist.py", line 42, in cb
            _drive(iterable, finished)
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/flat/twist.py", line 26, in _drive
            finished.callback('')
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 362, in callback
            self._startRunCallbacks(result)
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 545, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/appserver.py", line 181, in _cbFinishRender
            self.finishRequest(  True )
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/appserver.py", line 176, in finishRequest
            server.Request.finish(self)
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/web/http.py", line 866, in finish
            "Request.finish called on a request after its connection was lost; "
        exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.

Resolution: Handle early/exceptional TCP disconnects.

Related Issues:

• It looks like a closed ticket purports to fix this issue, at least for some cases: ​https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1366
• haproxy may behave unlike browsers. For instance, it may treat the connection to the gateway as Keep-Alive, even when the browser clients do not. I am using default configuration for this kind of behavior, and in generaly I expect haproxy to "play well" with many different http server implementations.