upload failed -- "I/O operation on closed file"

[zooko]

A call to tahoe backup on the command-line ended with this:

uploading './_darcs/patches/0000000452-203b744d7323110202fda6f42296ac24d699e79cfa82df294567276fda24c6ab'..                                                                                          [189/336]
 ./_darcs/patches/0000000452-203b744d7323110202fda6f42296ac24d699e79cfa82df294567276fda24c6ab -> URI:CHK:e2mh2nlllpjxqtihtadu2gdfuq:7274zdu36zq7m65svkkyp3eo3r65oeavmx6dbt7bsikoqwl3lpeq:1:1:332
uploading './_darcs/patches/0000000645-c5799a45bde6fb7b970e7a85b47121dbb72496f4e52ec927dadabb3e0c0ae46c'..
 ./_darcs/patches/0000000645-c5799a45bde6fb7b970e7a85b47121dbb72496f4e52ec927dadabb3e0c0ae46c -> URI:CHK:yzk4twdq36bmeslfb3us27l5zm:5rezjno7rdip2j7ics2w2zpwtylp4wsys55aymyrynplwzx7ztra:1:1:422
uploading './_darcs/patches/0000003657-20949c0699f1e7e939b33ec85764c2565e076988ff8cd51bdfba8b1f24133a15'..
Traceback (most recent call last):
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/runner.py", line 117, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/runner.py", line 102, in runner
    rc = cli.dispatch[command](so)
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/cli.py", line 569, in backup
    rc = tahoe_backup.backup(options)
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/tahoe_backup.py", line 325, in backup
    return bu.run()
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/tahoe_backup.py", line 118, in run
    new_backup_dircap = self.process(options.from_dir)
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/tahoe_backup.py", line 194, in process
    childcap, metadata = self.upload(childpath)
  File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/scripts/tahoe_backup.py", line 305, in upload
    raise HTTPError("Error during file PUT", resp)
HTTPError: Error during file PUT: 500 Internal Server Error
"Traceback (most recent call last):\x0a  File \"/home/zooko/playground/tahoe-lafs/dw/src/allmydata/util/pipeline.py\", line 65, in _ebDeferred\x0a    self.checkForFinished()\x0a  File \"/home/zooko/playground/tahoe-lafs/dw/src/allmydata/util/pipeline.py\", line 53, in checkForFinished\x0a    self.errback(self.failure)\x0a  File \"/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py\", line 391, in errback\x0a    self._startRunCallbacks(fail)\x0a  File \"/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py\", line 458, in _startRunCallbacks\x0a    self._runCallbacks()\x0a--- <exception caught here> ---\x0a  File \"/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py\", line 545, in _runCallbacks\x0a    current.result = callback(current.result, *args, **kw)\x0a  File \"/home/zooko/playground/tahoe-lafs/dw/src/allmydata/immutable/encode.py\", line 501, in _remove_shareholder\x0a    raise UploadUnhappinessError(msg)\x0aallmydata.interfaces.UploadUnhappinessError: shares could be placed or found on only 0 server(s). We were asked to place shares on at least 1 server(s) such that any 1 of them have enough shares to recover the file.: [Failure instance: Traceback (failure with no frames): <class 'allmydata.util.pipeline.PipelineError'>: <PipelineError error=([Failure instance: Traceback (failure with no frames): <class 'foolscap.ipb.DeadReferenceError'>: Connection was lost (to tubid=66ra) (during method=RIBucketWriter:write)\x0a])>\x0a]\x0a"

The twistd.log file had:

2012-07-17 08:09:46-0300 [-] 'Original exception:'
2012-07-17 08:09:46-0300 [-] Unhandled Error
        Traceback (most recent call last):
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 287, in addCallbacks
            self._runCallbacks()
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 545, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/immutable/upload.py", line 752, in <lambda>
            self._read_encrypted(length, ciphertext, hash_only, d2))
          File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/immutable/upload.py", line 767, in _read_encrypted
            d = defer.maybeDeferred(self.original.read, size)
        --- <exception caught here> ---
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 134, in maybeDeferred
            result = f(*args, **kw)
          File "/home/zooko/playground/tahoe-lafs/dw/src/allmydata/immutable/upload.py", line 1467, in read
            return defer.succeed([self._filehandle.read(length)])
        exceptions.ValueError: I/O operation on closed file
        
2012-07-17 08:09:46-0300 [-] Unhandled error in Deferred:
2012-07-17 08:09:46-0300 [-] Unhandled Error
        Traceback (most recent call last):
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/base.py", line 800, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/local/lib/python2.7/dist-packages/foolscap-0.6.3.post0-py2.7.egg/foolscap/eventual.py", line 26, in _turn
            cb(*args, **kwargs)
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 362, in callback
            self._startRunCallbacks(result)
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 458, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 545, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Nevow-0.10.0-py2.7.egg/nevow/appserver.py", line 91, in processingFailed
            request.finishRequest( False )
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Nevow-0.10.0-py2.7.egg/nevow/appserver.py", line 176, in finishRequest
            server.Request.finish(self)
          File "/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg/twisted/web/http.py", line 866, in finish
            "Request.finish called on a request after its connection was lost; "
        exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
        

The versions are:

$ tahoe --version-and-path
allmydata-tahoe: 1.9.0-r5601 (/home/zooko/playground/tahoe-lafs/dw/src)
foolscap: 0.6.3.post0 (/usr/local/lib/python2.7/dist-packages/foolscap-0.6.3.post0-py2.7.egg)
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958 (/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/pycryptopp-0.6.0.1206569328141510525648634803928199668821045408958-py2.7-linux-x86_64.egg)
zfec: 1.4.24 (/usr/local/lib/python2.7/dist-packages/zfec-1.4.24-py2.7-linux-x86_64.egg)
Twisted: 11.1.0 (/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-x86_64.egg)
Nevow: 0.10.0 (/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/Nevow-0.10.0-py2.7.egg)
zope.interface: unknown (/usr/local/lib/python2.7/dist-packages/zope.interface-3.6.1-py2.7-linux-x86_64.egg/zope)
python: 2.7.3 (/usr/bin/python)
platform: Linux-Ubuntu_12.04-x86_64-64bit_ELF (None)
pyOpenSSL: 0.12 (/usr/lib/python2.7/dist-packages)
simplejson: 2.3.2 (/usr/lib/python2.7/dist-packages)
pycrypto: 2.5 (/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/pycrypto-2.5-py2.7-linux-x86_64.egg)
pyasn1: unknown (/usr/lib/python2.7/dist-packages)
mock: 0.8.0beta3 (/usr/local/lib/python2.7/dist-packages)
setuptools: 0.6c16dev3 (/home/zooko/playground/tahoe-lafs/dw/support/lib/python2.7/site-packages/setuptools-0.6c16dev3.egg)