[tahoe-lafs-trac-stream] [Tahoe-LAFS] #1278: gateway won't serve any page; variety of interesting error messages in twistd.log

Tahoe-LAFS trac at tahoe-lafs.org
Wed Dec 3 21:10:05 UTC 2014 

#1278: gateway won't serve any page; variety of interesting error messages in
twistd.log
-------------------------+-------------------------------------------------
     Reporter:  zooko    |      Owner:  somebody
         Type:  defect   |     Status:  new
     Priority:  major    |  Milestone:  soon
    Component:  code     |    Version:  1.8.0
   Resolution:           |   Keywords:  reliability availability error hang
Launchpad Bug:           |  heisenbug wui anti-censorship test-needed
-------------------------+-------------------------------------------------

Comment (by zooko):

 Replying to [comment:7 daira]:
 > Zooko, what's the evidence that this is the same issue?

 First of all the observed behavior of the gateway is similar — goes slower
 and slower and then "locks ups" entirely, refusing to display even its own
 Welcome Page. But, like you say, actual stack traces are a more reliable
 indicator of a common cause, and we have them…

 WAG got these messages, in an incident report file named
 `incident-2014-11-18--08-59-32Z-nchatry.flog.bz2`:
 {{{
 06:20:45.186 [19723]: UNUSUAL error during GET FAILURE:
 [CopiedFailure instance: Traceback from remote host -- Traceback (most
 recent call last):
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 575, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/data/allmydata-tahoe-1.10.0/src/allmydata/mutable/retrieve.py",
 line 222, in <lambda>
     self._pause_deferred.addCallback(lambda ignored: d.callback(res))
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 380, in callback
     self._startRunCallbacks(result)
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 488, in _startRunCallbacks
     self._runCallbacks()
 --- <exception caught here> ---
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 575, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/data/allmydata-tahoe-1.10.0/src/allmydata/mutable/retrieve.py",
 line 228, in _check_for_stopped
     raise DownloadStopped("our Consumer called stopProducing()")
 allmydata.interfaces.DownloadStopped: our Consumer called stopProducing()
 ]
 }}}
 and:
 {{{
 08:59:33.278 [28364487]: Unhandled Error Traceback (most recent call
 last): Failure: exceptions.RuntimeError: Producer was not unregistered for
 /file/URI:MDMF:CENSORED:CENSORED FAILURE:
 [CopiedFailure instance: Traceback from remote host -- Traceback (most
 recent call last):
 Failure: exceptions.RuntimeError: Producer was not unregistered for
 /file/URI:MDMF:CENSORED:CENSORED
 ]
 }}}

 And these messages in an incident report file named `incident-2014-10-16--
 14-09-34Z-twxj47a.flog.bz2`:
 {{{
 2014-10-16_14:09:15.274918Z [11791490]: {'from-twisted':True,
 'incarnation':('\xd4\xf0\xeb\x82\x9b\xa3\x05$', None), 'level':20,
 'message':'Unhandled Error\nTraceback (most recent call last):\n File
 "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-...0.0/src/allmydata/web/filenode.py", line
 319, in replace_my_contents\n req.content.seek(0)\nexceptions.ValueError:
 I/O operation on closed file\n', 'system':'-', 'why':None, ...} FAILURE:
 [CopiedFailure instance: Traceback from remote host -- Traceback (most
 recent call last):
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 380, in callback
     self._startRunCallbacks(result)
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 488, in _startRunCallbacks
     self._runCallbacks()
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 575, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Nevow-0.11.1-py2.6.egg/nevow/appserver.py", line 163, in
 gotPageContext
     pageContext.tag.renderHTTP, pageContext
 --- <exception caught here> ---
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 137, in maybeDeferred
     result = f(*args, **kw)
   File "/data/allmydata-tahoe-1.10.0/src/allmydata/web/common.py", line
 352, in renderHTTP
     return m(ctx)
   File "/data/allmydata-tahoe-1.10.0/src/allmydata/web/filenode.py", line
 245, in render_PUT
     return self.replace_my_contents(req)
   File "/data/allmydata-tahoe-1.10.0/src/allmydata/web/filenode.py", line
 319, in replace_my_contents
     req.content.seek(0)
 exceptions.ValueError: I/O operation on closed file
 ]
 }}}
 and:
 {{{
 2014-10-16_14:09:15.281716Z [11791492]: {'from-twisted':True,
 'incarnation':('\xd4\xf0\xeb\x82\x9b\xa3\x05$', None), 'level':20,
 'message':'Unhandled Error\nTraceback (most recent call last):\n File
 "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Nevow-0.11.1-py2.6.eg...ost; "\nexceptions.RuntimeError:
 Request.finish called on a request after its connection was lost; use
 Request.notifyFinish to keep track of this.\n', 'system':'-', 'why':None,
 ...} FAILURE:
 [CopiedFailure instance: Traceback from remote host -- Traceback (most
 recent call last):
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Nevow-0.11.1-py2.6.egg/nevow/appserver.py", line 167, in
 gotPageContext
     processingFailed, self, pageContext
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 315, in addErrback
     errbackKeywords=kw)
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 293, in addCallbacks
     self._runCallbacks()
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86

 -- TRACEBACK ELIDED --

 -packages/Nevow-0.11.1-py2.6.egg/nevow/appserver.py", line 82, in
 processingFailed
     handler.renderHTTP_exception(ctx, reason)
   File "/data/allmydata-tahoe-1.10.0/src/allmydata/web/common.py", line
 329, in renderHTTP_exception
     return super.renderHTTP_exception(self, ctx, f)
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Nevow-0.11.1-py2.6.egg/nevow/appserver.py", line 61, in
 renderHTTP_exception
     request.finishRequest( False )
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Nevow-0.11.1-py2.6.egg/nevow/appserver.py", line 176, in
 finishRequest
     server.Request.finish(self)
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/server.py",
 line 231, in finish
     return http.Request.finish(self)
   File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line
 885, in finish
     "Request.finish called on a request after its connection was lost; "
 exceptions.RuntimeError: Request.finish called on a request after its
 connection was lost; use Request.notifyFinish to keep track of this.
 ]
 }}}

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1278#comment:8>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage

More information about the tahoe-lafs-trac-stream mailing list