Changes between Initial Version and Version 3 of Ticket #1794

Timestamp:

2014-11-28T16:01:57Z (10 years ago)

Author:

zooko

Comment:

My consulting client (codename "WAG", see comment:6:ticket:1278) has this symptom as well. This is from their storage node's twistd.log:

2014-10-16 19:37:01+0530 [-] Unhandled Error
        Traceback (most recent call last):
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 380, i
n callback
            self._startRunCallbacks(result)
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 488, i
n _startRunCallbacks
            self._runCallbacks()
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 575, i
n _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Nevow-0.11.1-py2.6.egg/nevow/appserver.py", line 163, in gotPageContext
            pageContext.tag.renderHTTP, pageContext
        --- <exception caught here> ---
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 137, i
n maybeDeferred
            result = f(*args, **kw)
          File "/data/allmydata-tahoe-1.10.0/src/allmydata/web/common.py", line 352, in renderHTTP
            return m(ctx)
          File "/data/allmydata-tahoe-1.10.0/src/allmydata/web/filenode.py", line 109, in render_PUT
            return self.replace_me_with_a_child(req, self.client, replace)
          File "/data/allmydata-tahoe-1.10.0/src/allmydata/web/filenode.py", line 32, in replace_me_with_a_child
            data = MutableFileHandle(req.content)
          File "/data/allmydata-tahoe-1.10.0/src/allmydata/mutable/publish.py", line 1219, in __init__
            self._filehandle.seek(0)
        exceptions.ValueError: I/O operation on closed file.
        

I think that this is caused by file descriptor resource exhaustion, because I've seen this in that twistd.log:

2014-11-20 13:33:36+0530 [HTTPChannel,143180,192.168.51.230] Unhandled Error
        Traceback (most recent call last):
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 88, in cal
lWithLogger
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 73, in cal
lWithContext
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 118, i
n callWithContext
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 81, in
 callWithContext
            
        --- <exception caught here> ---
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/posixbase.py", line 61
4, in _doReadOrWrite
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 215, in 
doRead
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 221, in 
_dataReceived
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py", line 581, 
in dataReceived
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 1609, in lin
eReceived
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 1702, in all
HeadersReceived
            
          File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 694, in gotL
ength
            
          File "/usr/lib64/python2.6/tempfile.py", line 475, in TemporaryFile
            
          File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner
            
        exceptions.OSError: [Errno 24] Too many open files: '/GFS2_tahoe/.tahoe-filestore1/tmp/tmpmKDgf6'