#1664 new defect

webapi fails to handle all TCP disconnects: "Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this."

Reported by: nejucomo Owned by: nobody
Priority: major Milestone: soon
Component: code-frontend-web Version: 1.9.0
Keywords: webapi error disconnect twisted Cc:
Launchpad Bug:

Description (last modified by nejucomo)

Symptoms: While running a gateway which has haproxy as a webapi client which I frequently restart, I see exceptions like this:

2012-01-21 14:35:15-0800 [-] Unhandled error in Deferred:
2012-01-21 14:35:15-0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/flat/twist.py", line 42, in cb
            _drive(iterable, finished)
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/flat/twist.py", line 26, in _drive
            finished.callback('')
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 362, in callback
            self._startRunCallbacks(result)
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 545, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/appserver.py", line 181, in _cbFinishRender
            self.finishRequest(  True )
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/appserver.py", line 176, in finishRequest
            server.Request.finish(self)
          File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/web/http.py", line 866, in finish
            "Request.finish called on a request after its connection was lost; "
        exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.

Resolution: Handle early/exceptional TCP disconnects.

Related Issues:

  • It looks like a closed ticket purports to fix this issue, at least for some cases: #1366 -also see: #1278
  • haproxy may behave unlike browsers. For instance, it may treat the connection to the gateway as Keep-Alive, even when the browser clients do not. I am using default configuration for this kind of behavior, and in generaly I expect haproxy to "play well" with many different http server implementations.

Change History (6)

comment:1 Changed at 2012-01-22T00:11:53Z by nejucomo

  • Description modified (diff)

comment:2 Changed at 2012-03-12T19:27:24Z by davidsarah

  • Component changed from unknown to code-frontend-web
  • Keywords webapi error disconnect twisted added

comment:3 Changed at 2012-04-01T23:07:29Z by davidsarah

  • Milestone changed from undecided to 1.11.0

comment:4 follow-up: Changed at 2012-04-02T03:24:25Z by nejucomo

After some more experience with twisted.web, this seems to be a very common bug. For instance, this query shows twisted made the same mistake internally in three different components:

Twisted trac query for "Request.finish called on a request after its connection was lost"

comment:5 Changed at 2012-04-02T03:31:35Z by nejucomo

I am somewhat confident this behavior is benign; the web server will continue processing other requests without being affected. For this reason I'd lower the priority.

There's only two possible drawbacks I now brainstorm:

  • The exception may scare users.
  • The process generating the response may be wasting resources; especially if this is a very long-running request which consumes many resources.

There *could* be a bug if a failure for the client to receive a request result leads to corruption. I doubt that is the case, at least for the RESTful storage APIs.

comment:6 in reply to: ↑ 4 Changed at 2012-04-02T18:00:41Z by davidsarah

Replying to nejucomo:

For instance, this query shows twisted made the same mistake internally in three different components [...]

Actually at least five; the query didn't catch https://twistedmatrix.com/trac/ticket/4901 or https://twistedmatrix.com/trac/ticket/5132 (found by searching for just notifyFinish).

Note: See TracTickets for help on using tickets.