Context Navigation

← Previous Ticket
Next Ticket →

Opened at 2013-11-26T04:32:53Z

Closed at 2014-04-15T00:49:57Z

#2116 closed defect (fixed)

xml parse error from S4 service

Reported by:	zooko	Owned by:	daira
Priority:	major	Milestone:	undecided
Component:	code-storage	Version:	1.9.0-s3branch
Keywords:	s3 error txaws retry LeastAuthority	Cc:
Launchpad Bug:

Description (last modified by daira)

zooko@spark ~/research/papers $ tahoe backup . $WRITABLE_DIRCAP:
Traceback (most recent call last):
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/cli.py", line 574, in backup
    rc = tahoe_backup.backup(options)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 325, in backup
    return bu.run()
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 118, in run
    new_backup_dircap = self.process(options.from_dir)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 194, in process
    childcap, metadata = self.upload(childpath)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 305, in upload
    raise HTTPError("Error during file PUT", resp)
HTTPError: Error during file PUT: 500 Internal Server Error
"Traceback (most recent call last):
  File \"/home/zooko/playground/LAFS/pristine/support/lib/python2.7/site-packages/foolscap-0.6.4-py2.7.egg/foolscap/call.py\", line 753, in receiveClose
    self.request.fail(f)
  File \"/home/zooko/playground/LAFS/pristine/support/lib/python2.7/site-packages/foolscap-0.6.4-py2.7.egg/foolscap/call.py\", line 95, in fail
    self.deferred.errback(why)
  File \"/home/zooko/.local/lib/python2.7/site-packages/twisted/internet/defer.py\", line 421, in errback
    self._startRunCallbacks(fail)
  File \"/home/zooko/.local/lib/python2.7/site-packages/twisted/internet/defer.py\", line 488, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File \"/home/zooko/.local/lib/python2.7/site-packages/twisted/internet/defer.py\", line 575, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"/home/zooko/playground/LAFS/pristine/src/allmydata/immutable/upload.py\", line 604, in _got_response
    return self._loop()
  File \"/home/zooko/playground/LAFS/pristine/src/allmydata/immutable/upload.py\", line 516, in _loop
    return self._failed(msg)
  File \"/home/zooko/playground/LAFS/pristine/src/allmydata/immutable/upload.py\", line 617, in _failed
    raise UploadUnhappinessError(msg)
allmydata.interfaces.UploadUnhappinessError: server selection failed for 
<Tahoe2ServerSelector for upload qd7vq>: shares could be placed or found on 
only 0 server(s). We were asked to place shares on at least 1 server(s) such 
that any 1 of them have enough shares to recover the file. (placed 0 shares out 
of 1 total (1 homeless), want to place shares on at least 1 servers such that 
any 1 of them have enough shares to recover the file, sent 1 queries to 1 
servers, 0 queries placed some shares, 1 placed none (of which 0 placed none 
due to the server being full and 1 placed none due to an error)) (last failure 
(from <ServerTracker for server 6dwd2q and SI qd7vq>) was:
[Failure instance: Traceback (failure with no frames): <class 'foolscap.tokens.RemoteException'>:
<RemoteException around '[CopiedFailure instance: Traceback from remote host -- Traceback (most recent call last):
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/internet/tcp.py\", line 287, in connectionLost
    protocol.connectionLost(reason)
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/web/client.py\", line 193, in connectionLost
    self.factory._disconnectedDeferred.callback(None)
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 368, in callback
    self._startRunCallbacks(result)
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 464, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> --- 
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"/usr/local/lib/python2.7/dist-packages/txAWS-0.2.1.post4-py2.7.egg/txaws/s3/client.py\", line 153, in _parse_get_bucket
    root = XML(xml_bytes)
  File \"/usr/local/lib/python2.7/dist-packages/txAWS-0.2.1.post4-py2.7.egg/txaws/util.py\", line 61, in XML
    return parser.close()
  File \"/usr/lib/python2.7/xml/etree/ElementTree.py\", line 1655, in close
    self._raiseerror(v)
  File \"/usr/lib/python2.7/xml/etree/ElementTree.py\", line 1507, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
]'>
])
"

Change History (29)

comment:1 Changed at 2013-11-26T04:34:59Z by zooko

I tried the same command-line again, and this time it worked. I had started by tahoe-lafs gateway a few second before I got this xml parse error, so I suspect that this is a case of #719.

comment:2 Changed at 2013-11-26T17:00:40Z by daira

I don't think it is a case of #719: the S3 container code in the server got as far as making an S3 request and trying to parse the result, which was not valid XML. #719 refers to the case where the client can't reach enough servers.

Last edited at 2013-11-26T19:18:17Z by daira (previous) (diff)

comment:3 Changed at 2013-11-26T17:01:10Z by daira

Actually the response from S3 seems to have been empty.

comment:4 follow-up: ↓ 6 Changed at 2013-11-26T17:08:41Z by daira

Keywords s3 error txaws retry LeastAuthority added
Owner set to daira
Status changed from new to assigned
Version changed from 1.10.0 to 1.9.0-s3branch

Hmm, maybe an empty S3 response should trigger a retry?

comment:5 Changed at 2013-11-26T17:13:13Z by daira

Sigh, I wish we could reduce that error report to something more concise -- although I suppose we should be grateful that the actual error was in there somewhere.

Last edited at 2013-11-26T17:13:26Z by daira (previous) (diff)

comment:6 in reply to: ↑ 4 Changed at 2013-11-27T19:59:27Z by zooko

Replying to daira:

Hmm, maybe an empty S3 response should trigger a retry?

I guess. :-/

This is what I think of as "snowshoes". Having our upper layer perform some redundancy or retrying just to work-around what appear to be bugs or unreliability from the lower layer. Ick.

comment:7 follow-up: ↓ 8 Changed at 2013-11-27T20:01:57Z by daira

Well, I'm philosophically resistant to that as well, but I have no confidence that we can get the underlying problem with S3 fixed.

comment:8 in reply to: ↑ 7 Changed at 2013-11-27T20:03:38Z by zooko

Replying to daira:

Well, I'm philosophically resistant to that as well, but I have no confidence that we can get the underlying problem with S3 fixed.

Yeah, I agree that we should go ahead and add a limited number of exponential-backoff-retries for this.

comment:9 Changed at 2013-11-27T20:04:42Z by daira

Note that this isn't the first time I've seen an empty response from S3. They are rare, but they've come up before - we just haven't recorded that problem until now.

comment:10 Changed at 2013-11-28T12:52:35Z by zooko

I just a similar failure again:

Traceback (most recent call last):
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/cli.py", line 574, in backup
    rc = tahoe_backup.backup(options)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 325, in backup
    return bu.run()
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 118, in run
    new_backup_dircap = self.process(options.from_dir)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 194, in process
    childcap, metadata = self.upload(childpath)
  File "/home/zooko/playground/LAFS/pristine/src/allmydata/scripts/tahoe_backup.py", line 305, in upload
    raise HTTPError("Error during file PUT", resp)
HTTPError: Error during file PUT: 500 Internal Server Error
"Traceback (most recent call last):
  File \"/home/zooko/playground/LAFS/pristine/support/lib/python2.7/site-packages/foolscap-0.6.4-py2.7.egg/foolscap/call.py\", line 753, in receiveClose
    self.request.fail(f)
  File \"/home/zooko/playground/LAFS/pristine/support/lib/python2.7/site-packages/foolscap-0.6.4-py2.7.egg/foolscap/call.py\", line 95, in fail
    self.deferred.errback(why)
  File \"/home/zooko/.local/lib/python2.7/site-packages/twisted/internet/defer.py\", line 421, in errback
    self._startRunCallbacks(fail)
  File \"/home/zooko/.local/lib/python2.7/site-packages/twisted/internet/defer.py\", line 488, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File \"/home/zooko/.local/lib/python2.7/site-packages/twisted/internet/defer.py\", line 575, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"/home/zooko/playground/LAFS/pristine/src/allmydata/immutable/upload.py\", line 604, in _got_response
    return self._loop()
  File \"/home/zooko/playground/LAFS/pristine/src/allmydata/immutable/upload.py\", line 516, in _loop
    return self._failed(msg)
  File \"/home/zooko/playground/LAFS/pristine/src/allmydata/immutable/upload.py\", line 617, in _failed
    raise UploadUnhappinessError(msg)
allmydata.interfaces.UploadUnhappinessError: server selection failed for <Tahoe2ServerSelector 
for upload s7uqw>: shares could be placed or found on only 0 server(s). We were asked to place 
shares on at least 1 server(s) such that any 1 of them have enough shares to recover the file. 
(placed 0 shares out of 1 total (1 homeless), want to place shares on at least 1 servers such 
that any 1 of them have enough shares to recover the file, sent 1 queries to 1 servers, 0 
queries placed some shares, 1 placed none (of which 0 placed none due to the server being full 
and 1 placed none due to an error)) (last failure (from <ServerTracker for server 6dwd2q and SI 
s7uqw>) was:
[Failure instance: Traceback (failure with no frames): <class 'foolscap.tokens.RemoteException'>: <RemoteException around '[CopiedFailure instance: Traceback from remote host -- Traceback (most recent call last):
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/internet/tcp.py\", line 287, in connectionLost
    protocol.connectionLost(reason)
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/web/client.py\", line 193, in connectionLost
    self.factory._disconnectedDeferred.callback(None)
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 368, in callback
    self._startRunCallbacks(result)
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 464, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File \"/home/customer/LAFS_source/support/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"/usr/local/lib/python2.7/dist-packages/txAWS-0.2.1.post4-py2.7.egg/txaws/s3/client.py\", line 153, in _parse_get_bucket
    root = XML(xml_bytes)
  File \"/usr/local/lib/python2.7/dist-packages/txAWS-0.2.1.post4-py2.7.egg/txaws/util.py\", line 61, in XML
    return parser.close()
  File \"/usr/lib/python2.7/xml/etree/ElementTree.py\", line 1655, in close
    self._raiseerror(v)
  File \"/usr/lib/python2.7/xml/etree/ElementTree.py\", line 1507, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
]'>
])
"

Last edited at 2014-03-17T22:00:44Z by daira (previous) (diff)

comment:11 Changed at 2013-11-29T05:42:21Z by zooko

Hm, I seem to get this every time that I run a full tahoe backup on my data now. ☹ It seems like the LeastAuthority S4 service is suffering badly from this.

  File \"/usr/local/lib/python2.7/dist-packages/txAWS-0.2.1.post4-py2.7.egg/txaws/s3/client.py\", line 153, in _parse_get_bucket
    root = XML(xml_bytes)
  File \"/usr/local/lib/python2.7/dist-packages/txAWS-0.2.1.post4-py2.7.egg/txaws/util.py\", line 61, in XML
    return parser.close()
  File \"/usr/lib/python2.7/xml/etree/ElementTree.py\", line 1655, in close
    self._raiseerror(v)
  File \"/usr/lib/python2.7/xml/etree/ElementTree.py\", line 1507, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
]'>
])
"

comment:12 Changed at 2013-11-30T20:55:30Z by daira

Priority changed from normal to major

comment:13 Changed at 2013-12-01T16:21:22Z by zooko

I just got this same error again, but this time on read instead of on upload. Manually retrying a couple of times (by hitting C-r in my web browser) got it to work.

comment:14 Changed at 2014-03-04T19:29:52Z by daira

CyberAxe just saw the same error: http://sebsauvage.net/paste/?e301277924eb38dc#6W5WPJkf1HgW3+rdFVvQ/2BqzYIZ0ZwQyPUb08+kY0A=

Last edited at 2014-03-04T19:30:09Z by daira (previous) (diff)

comment:15 Changed at 2014-03-04T19:30:43Z by daira

I'm going to make it retry.

comment:16 follow-up: ↓ 17 Changed at 2014-03-10T21:45:28Z by daira

Note that the error occurs in txAWS when it tries to parse an empty response body for a 500 error as XML. The fix might need to be in txAWS as well.

It's a bit difficult to test this, because the tests intentionally do not depend on txAWS (currently they only test the mock cloud backend, not the service container implementations).

comment:17 in reply to: ↑ 16 Changed at 2014-03-11T15:55:09Z by daira

Replying to daira:

Note that the error occurs in txAWS when it tries to parse an empty response body for a 500 error as XML.

No, this is a wrong inference. The error occurs in txaws.s3.client._parse_get_bucket, which should not be reached if the S3 server responded with a 500 error. (The 500 error in the stack trace is from the Tahoe-LAFS gateway, not S3.) So it looks like S3 returned success with an empty body.

comment:18 Changed at 2014-03-17T22:03:07Z by daira

Description modified (diff)

comment:19 Changed at 2014-03-17T22:23:31Z by daira

Pasting the definitions from txaws.util into a Python interactive session:

$ python
Python 2.7.5+ (default, Jun  2 2013, 13:26:34) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> from xml.etree.ElementTree import XMLTreeBuilder
>>> def XML(text):
...     parser = NamespaceFixXmlTreeBuilder()
...     parser.feed(text)
...     return parser.close()
... 
>>> class NamespaceFixXmlTreeBuilder(XMLTreeBuilder):
...     def _fixname(self, key):
...         if "}" in key:
...             key = key.split("}", 1)[1]
...         return key
... 
>>> XML("")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in XML
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1654, in close
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

Last edited at 2014-03-17T22:26:38Z by daira (previous) (diff)

comment:20 Changed at 2014-03-17T22:25:24Z by daira

It looks like only an empty string will give the xml.etree.ElementTree.ParseError: no element found: line 1, column 0 error.

comment:21 Changed at 2014-03-18T17:00:13Z by CyberAxe

Got this last night, did Za do the update on my backend? If so it didn't work.

:: ---------------------  START of LOG  ------------ 
:: START Date: 20141803	Time:  24638 
:: Backup-Now assuming tahoe: as alias and BackedUp as Target Location in Grid 
:: c:\Python27\ must be in system PATH: see tahoe-lafs Windows wiki for help 
:: Will Backup everything in folder c:\BackMeUp\ 
:: v0040  
:: ---------------------  Start Backup  ------------ 
:: ---------------------  Errors Backup  ----------- 
Traceback (most recent call last):
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\cli.py", line 574, in backup
    rc = tahoe_backup.backup(options)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 325, in backup
    return bu.run()
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 118, in run
    new_backup_dircap = self.process(options.from_dir)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 194, in process
    childcap, metadata = self.upload(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 305, in upload
    raise HTTPError("Error during file PUT", resp)
HTTPError: Error during file PUT: 500 Internal Server Error
"Traceback (most recent call last):
  File \"c:\\allmydata-tahoe-1.10.0\\src\\allmydata\\util\\pipeline.py\", line 65, in _ebDeferred
    self.checkForFinished()
  File \"c:\\allmydata-tahoe-1.10.0\\src\\allmydata\\util\\pipeline.py\", line 53, in checkForFinished
    self.errback(self.failure)
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 422, in errback
    self._startRunCallbacks(fail)
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 489, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 576, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"c:\\allmydata-tahoe-1.10.0\\src\\allmydata\\immutable\\encode.py\", line 501, in _remove_shareholder
    raise UploadUnhappinessError(msg)
allmydata.interfaces.UploadUnhappinessError: shares could be placed or found on only 0 server(s). We were asked to place shares on at least 1 server(s) such that any 1 of them have enough shares to recover the file.: [Failure instance: Traceback (failure with no frames): <class 'allmydata.util.pipeline.PipelineError'>: <PipelineError error=([Failure instance: Traceback (failure with no frames): <class 'foolscap.tokens.RemoteException'>: <RemoteException around '[CopiedFailure instance: Traceback from remote host -- Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.
]'>
])>
]
"
:: ---------------------  End Backup  --------------

Last edited at 2014-03-18T17:07:30Z by zooko (previous) (diff)

comment:22 follow-up: ↓ 28 Changed at 2014-03-21T18:21:35Z by CyberAxe

:: --------------------- START of LOG ------------ :: START Date: 20142103 Time: 21327 :: Backup-Now assuming tahoe: as alias and BackedUp? as Target Location in Grid :: c:\Python27\ must be in system PATH: see tahoe-lafs Windows wiki for help :: Will Backup everything in folder c:\BackMeUp?\ :: v0040 :: --------------------- Start Backup ------------ :: --------------------- Errors Backup -----------

Traceback (most recent call last):
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\cli.py", line 574, in backup
    rc = tahoe_backup.backup(options)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 325, in backup
    return bu.run()
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 118, in run
    new_backup_dircap = self.process(options.from_dir)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 194, in process
    childcap, metadata = self.upload(childpath)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 305, in upload
    raise HTTPError("Error during file PUT", resp)
HTTPError: Error during file PUT: 500 Internal Server Error
"Traceback (most recent call last):
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\foolscap-0.6.4-py2.7.egg\\foolscap\\call.py\", line 753, in receiveClose
    self.request.fail(f)
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\foolscap-0.6.4-py2.7.egg\\foolscap\\call.py\", line 95, in fail
    self.deferred.errback(why)
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 422, in errback
    self._startRunCallbacks(fail)
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 489, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 576, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"c:\\allmydata-tahoe-1.10.0\\src\\allmydata\\immutable\\upload.py\", line 604, in _got_response
    return self._loop()
  File \"c:\\allmydata-tahoe-1.10.0\\src\\allmydata\\immutable\\upload.py\", line 516, in _loop
    return self._failed(msg)
  File \"c:\\allmydata-tahoe-1.10.0\\src\\allmydata\\immutable\\upload.py\", line 617, in _failed
    raise UploadUnhappinessError(msg)
allmydata.interfaces.UploadUnhappinessError: server selection failed for <Tahoe2ServerSelector for upload qtquq>: shares could be placed or found on only 0 server(s). We were asked to place shares on at least 1 server(s) such that any 1 of them have enough shares to recover the file. (placed 0 shares out of 1 total (1 homeless), want to place shares on at least 1 servers such that any 1 of them have enough shares to recover the file, sent 1 queries to 1 servers, 0 queries placed some shares, 1 placed none (of which 0 placed none due to the server being full and 1 placed none due to an error)) (last failure (from <ServerTracker for server xfia4afr and SI qtquq>) was: [Failure instance: Traceback (failure with no frames): <class 'foolscap.tokens.RemoteException'>: <RemoteException around '[CopiedFailure instance: Traceback from remote host -- Traceback (most recent call last):
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/tcp.py\", line 299, in connectionLost
    protocol.connectionLost(reason)
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/web/client.py\", line 204, in connectionLost
    self.factory._disconnectedDeferred.callback(None)
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 382, in callback
    self._startRunCallbacks(result)
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 490, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 577, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"/home/customer/LAFS_source/src/allmydata/storage/backends/cloud/cloud_common.py\", line 370, in _retry
    d2 = self._handle_error(f, 1, None, description, operation, *args, **kwargs)
  File \"/home/customer/LAFS_source/src/allmydata/storage/backends/cloud/cloud_common.py\", line 417, in _handle_error
    d = task.deferLater(self._reactor, BACKOFF_SECONDS_BEFORE_RETRY[trynum-1], operation, *args, **kwargs)
exceptions.AttributeError: S3Container instance has no attribute '_reactor'
]'>
])
"
:: ---------------------  End Backup  -------------- 
:: ---------------------  Start View Latest  ------- 
dr-x    - Mar 14 19:50                                                    Joe
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (10) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (10).xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (100) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (101) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (102) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (103) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (104) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (105) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (106) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (107) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (108) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (109) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (11) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (11).xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (110) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (111) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (112) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (113) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (114) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (115) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (116) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (117) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (118) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (119) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (12) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (12).xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (120) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (121) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (122) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (123) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (124) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (125) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (126) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (127) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (128) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (129) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (13) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (13).xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (130) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (131) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (132) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (133) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (134) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (135) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (136) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (137) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (138) - Copy.xlsx
-r-- 6182 Mar 14 19:49 New Microsoft Excel Worksheet - Copy (139) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (14) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (14).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (15) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (15).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (16) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (16).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (17) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (17).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (18) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (18).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (19) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (19).xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy (2) - Copy.xlsx
-r-- 6182 Mar 14 19:49          New Microsoft Excel Worksheet - Copy (2).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (20) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (20).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (21) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (21).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (22) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (22).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (23) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (23).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (24) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (24).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (25) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (25).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (26) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (26).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (27) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (27).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (28) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (28).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (29) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (29).xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy (3) - Copy.xlsx
-r-- 6182 Mar 14 19:49          New Microsoft Excel Worksheet - Copy (3).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (30) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (30).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (31) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (31).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (32) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (32).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (33) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (33).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (34) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (34).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (35) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (35).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (36) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (36).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (37) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (37).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (38) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (38).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (39) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (39).xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy (4) - Copy.xlsx
-r-- 6182 Mar 14 19:49          New Microsoft Excel Worksheet - Copy (4).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (40) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (40).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (41) - Copy.xlsx
-r-- 6182 Mar 14 19:49         New Microsoft Excel Worksheet - Copy (41).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (42) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (43) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (44) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (45) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (46) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (47) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (48) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (49) - Copy.xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy (5) - Copy.xlsx
-r-- 6182 Mar 14 19:49          New Microsoft Excel Worksheet - Copy (5).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (50) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (51) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (52) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (53) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (54) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (55) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (56) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (57) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (58) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (59) - Copy.xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy (6) - Copy.xlsx
-r-- 6182 Mar 14 19:49          New Microsoft Excel Worksheet - Copy (6).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (60) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (61) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (62) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (63) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (64) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (65) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (66) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (67) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (68) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (69) - Copy.xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy (7) - Copy.xlsx
-r-- 6182 Mar 14 19:49          New Microsoft Excel Worksheet - Copy (7).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (70) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (71) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (72) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (73) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (74) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (75) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (76) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (77) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (78) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (79) - Copy.xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy (8) - Copy.xlsx
-r-- 6182 Mar 14 19:49          New Microsoft Excel Worksheet - Copy (8).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (80) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (81) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (82) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (83) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (84) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (85) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (86) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (87) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (88) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (89) - Copy.xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy (9) - Copy.xlsx
-r-- 6182 Mar 14 19:49          New Microsoft Excel Worksheet - Copy (9).xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (90) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (91) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (92) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (93) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (94) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (95) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (96) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (97) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (98) - Copy.xlsx
-r-- 6182 Mar 14 19:49  New Microsoft Excel Worksheet - Copy (99) - Copy.xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy - Copy (2).xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy - Copy (3).xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy - Copy (4).xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy - Copy (5).xlsx
-r-- 6182 Mar 14 19:49   New Microsoft Excel Worksheet - Copy - Copy (6).xlsx
-r-- 6182 Mar 14 19:49       New Microsoft Excel Worksheet - Copy - Copy.xlsx
-r-- 6182 Mar 14 19:49              New Microsoft Excel Worksheet - Copy.xlsx
-r-- 6182 Mar 14 19:49                     New Microsoft Excel Worksheet.xlsx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (10) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (10).docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (100) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (101) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (102) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (103) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (104) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (105) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (106) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (107) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (108) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (109) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (11) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (11).docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (110) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (111) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (112) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (113) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (114) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (115) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (116) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (117) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (118) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (119) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (12) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (12).docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (120) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (121) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (122) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (123) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (124) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (125) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (126) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (127) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (128) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (129) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (13) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (13).docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (130) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (131) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (132) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (133) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (134) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (135) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (136) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (137) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (138) - Copy.docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (139) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (14) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (14).docx
-r--    0 Mar 14 19:49   New Microsoft Word Document - Copy (140) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (15) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (15).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (16) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (16).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (17) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (17).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (18) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (18).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (19) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (19).docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy (2) - Copy.docx
-r--    0 Mar 14 19:49            New Microsoft Word Document - Copy (2).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (20) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (20).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (21) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (21).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (22) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (22).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (23) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (23).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (24) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (24).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (25) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (25).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (26) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (26).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (27) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (27).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (28) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (28).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (29) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (29).docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy (3) - Copy.docx
-r--    0 Mar 14 19:49            New Microsoft Word Document - Copy (3).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (30) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (30).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (31) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (31).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (32) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (32).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (33) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (33).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (34) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (34).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (35) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (35).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (36) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (36).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (37) - Copy.docx
-r--    0 Mar 14 19:49           New Microsoft Word Document - Copy (37).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (38) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (39) - Copy.docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy (4) - Copy.docx
-r--    0 Mar 14 19:49            New Microsoft Word Document - Copy (4).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (40) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (41) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (42) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (43) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (44) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (45) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (46) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (47) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (48) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (49) - Copy.docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy (5) - Copy.docx
-r--    0 Mar 14 19:49            New Microsoft Word Document - Copy (5).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (50) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (51) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (52) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (53) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (54) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (55) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (56) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (57) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (58) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (59) - Copy.docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy (6) - Copy.docx
-r--    0 Mar 14 19:49            New Microsoft Word Document - Copy (6).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (60) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (61) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (62) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (63) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (64) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (65) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (66) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (67) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (68) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (69) - Copy.docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy (7) - Copy.docx
-r--    0 Mar 14 19:49            New Microsoft Word Document - Copy (7).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (70) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (71) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (72) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (73) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (74) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (75) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (76) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (77) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (78) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (79) - Copy.docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy (8) - Copy.docx
-r--    0 Mar 14 19:49            New Microsoft Word Document - Copy (8).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (80) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (81) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (82) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (83) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (84) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (85) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (86) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (87) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (88) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (89) - Copy.docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy (9) - Copy.docx
-r--    0 Mar 14 19:49            New Microsoft Word Document - Copy (9).docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (90) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (91) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (92) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (93) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (94) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (95) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (96) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (97) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (98) - Copy.docx
-r--    0 Mar 14 19:49    New Microsoft Word Document - Copy (99) - Copy.docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy - Copy (2).docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy - Copy (3).docx
-r--    0 Mar 14 19:49     New Microsoft Word Document - Copy - Copy (4).docx
-r--    0 Mar 14 19:49         New Microsoft Word Document - Copy - Copy.docx
-r--    0 Mar 14 19:49                New Microsoft Word Document - Copy.docx
-r--    0 Mar 14 19:49                       New Microsoft Word Document.docx
dr-x    - Mar 14 19:49                                                   Will
-r--  861 Mar 14 19:49                         bob - Copy (10) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (10).contact
-r--  861 Mar 14 19:49                        bob - Copy (100) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (101) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (102) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (103) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (104) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (105) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (106) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (107) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (108) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (109) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (11) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (11).contact
-r--  861 Mar 14 19:49                        bob - Copy (110) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (111) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (112) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (113) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (114) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (115) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (116) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (117) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (118) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (119) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (12) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (12).contact
-r--  861 Mar 14 19:49                        bob - Copy (120) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (121) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (122) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (123) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (124) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (125) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (126) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (127) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (128) - Copy.contact
-r--  861 Mar 14 19:49                        bob - Copy (129) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (13) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (13).contact
-r--  861 Mar 14 19:49                         bob - Copy (14) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (14).contact
-r--  861 Mar 14 19:49                         bob - Copy (15) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (15).contact
-r--  861 Mar 14 19:49                         bob - Copy (16) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (16).contact
-r--  861 Mar 14 19:49                         bob - Copy (17) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (17).contact
-r--  861 Mar 14 19:49                         bob - Copy (18) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (18).contact
-r--  861 Mar 14 19:49                         bob - Copy (19) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (19).contact
-r--  861 Mar 14 19:49                          bob - Copy (2) - Copy.contact
-r--  861 Mar 14 19:49                                 bob - Copy (2).contact
-r--  861 Mar 14 19:49                         bob - Copy (20) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (20).contact
-r--  861 Mar 14 19:49                         bob - Copy (21) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (21).contact
-r--  861 Mar 14 19:49                         bob - Copy (22) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (22).contact
-r--  861 Mar 14 19:49                         bob - Copy (23) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (23).contact
-r--  861 Mar 14 19:49                         bob - Copy (24) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (24).contact
-r--  861 Mar 14 19:49                         bob - Copy (25) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (25).contact
-r--  861 Mar 14 19:49                         bob - Copy (26) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (26).contact
-r--  861 Mar 14 19:49                         bob - Copy (27) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (27).contact
-r--  861 Mar 14 19:49                         bob - Copy (28) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (28).contact
-r--  861 Mar 14 19:49                         bob - Copy (29) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (29).contact
-r--  861 Mar 14 19:49                          bob - Copy (3) - Copy.contact
-r--  861 Mar 14 19:49                                 bob - Copy (3).contact
-r--  861 Mar 14 19:49                         bob - Copy (30) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (30).contact
-r--  861 Mar 14 19:49                         bob - Copy (31) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (31).contact
-r--  861 Mar 14 19:49                         bob - Copy (32) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (32).contact
-r--  861 Mar 14 19:49                         bob - Copy (33) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (33).contact
-r--  861 Mar 14 19:49                         bob - Copy (34) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (34).contact
-r--  861 Mar 14 19:49                         bob - Copy (35) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (35).contact
-r--  861 Mar 14 19:49                         bob - Copy (36) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (36).contact
-r--  861 Mar 14 19:49                         bob - Copy (37) - Copy.contact
-r--  861 Mar 14 19:49                                bob - Copy (37).contact
-r--  861 Mar 14 19:49                         bob - Copy (38) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (39) - Copy.contact
-r--  861 Mar 14 19:49                          bob - Copy (4) - Copy.contact
-r--  861 Mar 14 19:49                                 bob - Copy (4).contact
-r--  861 Mar 14 19:49                         bob - Copy (40) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (41) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (42) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (43) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (44) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (45) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (46) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (47) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (48) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (49) - Copy.contact
-r--  861 Mar 14 19:49                          bob - Copy (5) - Copy.contact
-r--  861 Mar 14 19:49                                 bob - Copy (5).contact
-r--  861 Mar 14 19:49                         bob - Copy (50) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (51) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (52) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (53) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (54) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (55) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (56) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (57) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (58) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (59) - Copy.contact
-r--  861 Mar 14 19:49                          bob - Copy (6) - Copy.contact
-r--  861 Mar 14 19:49                                 bob - Copy (6).contact
-r--  861 Mar 14 19:49                         bob - Copy (60) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (61) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (62) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (63) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (64) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (65) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (66) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (67) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (68) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (69) - Copy.contact
-r--  861 Mar 14 19:49                          bob - Copy (7) - Copy.contact
-r--  861 Mar 14 19:49                                 bob - Copy (7).contact
-r--  861 Mar 14 19:49                         bob - Copy (70) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (71) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (72) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (73) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (74) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (75) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (76) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (77) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (78) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (79) - Copy.contact
-r--  861 Mar 14 19:49                          bob - Copy (8) - Copy.contact
-r--  861 Mar 14 19:49                                 bob - Copy (8).contact
-r--  861 Mar 14 19:49                         bob - Copy (80) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (81) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (82) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (83) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (84) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (85) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (86) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (87) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (88) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (89) - Copy.contact
-r--  861 Mar 14 19:49                          bob - Copy (9) - Copy.contact
-r--  861 Mar 14 19:49                                 bob - Copy (9).contact
-r--  861 Mar 14 19:49                         bob - Copy (90) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (91) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (92) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (93) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (94) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (95) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (96) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (97) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (98) - Copy.contact
-r--  861 Mar 14 19:49                         bob - Copy (99) - Copy.contact
-r--  861 Mar 14 19:49                          bob - Copy - Copy (2).contact
-r--  861 Mar 14 19:49                          bob - Copy - Copy (3).contact
-r--  861 Mar 14 19:49                          bob - Copy - Copy (4).contact
-r--  861 Mar 14 19:49                              bob - Copy - Copy.contact
-r--  861 Mar 14 19:49                                     bob - Copy.contact
-r--  861 Mar 14 19:48                                            bob.contact
dr-x    - Mar 14 19:50                                                   logs}}}
:: ---------------------  End View Latest  --------- 
:: ---------------------  Start Deep Check   ------- 
:: SET TO RUN  
:: ---------------------  End Deep Check   --------- 
:: ---------------------  Start Grid Stats  -------- 
:: SET TO RUN  
:: ---------------------  End Grid Stats   --------- 
:: ---------------------  Start Log Backup   ------- 
:: Copy local log files to future backups to GRID 
:: ---------------------  END Log Backup     ------- 
:: END Date: Fri 03/21/2014	Time:  2:49:27.69 
:: ---------------------  END of LOG  --------------

Version 1, edited at 2014-03-21T18:24:21Z by CyberAxe (previous) (next) (diff)

comment:23 follow-up: ↓ 26 Changed at 2014-03-21T21:13:23Z by CyberAxe

(3:10:26 PM) CyberAxe?: Seems like instead of the 500 error now I'm getting it just stuck over and over, maybe I'm not waitting long enought for the 500 error to generate but I've waited hours. http://sebsauvage.net/paste/?f38e2b33f87e1422#A8iAnduHoydJrE6XejuwYX6CHoRx/iWBxUb+OLx1mvM= I have to terminate the process here's what it says when I do that. (3:12:20 PM) CyberAxe?: the last line it's stuck on in this example is a create folder action, when I restart backup it does use that folder, so it is creating it re-using old directory for 'c:\BackMeUp?\WesternColorado_files\images\WaspWar3_files\9' (3:12:20 PM) CyberAxe?: re-using old directory for 'c:\BackMeUp?\WesternColorado_files\images\WaspWar3_files' (3:12:20 PM) CyberAxe?: re-using old directory for 'c:\BackMeUp?\WesternColorado_files\images' (3:12:20 PM) CyberAxe?: re-using old directory for 'c:\BackMeUp?\WesternColorado_files' (3:12:20 PM) CyberAxe?: re-using old directory for 'c:\BackMeUp?\' <----- where it got stuck.

comment:24 Changed at 2014-03-22T02:23:07Z by CyberAxe

Here's another trackback error I got after the backend update to fix 500 error

 File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 422, in errback
    self._startRunCallbacks(fail)
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 489, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File \"C:\\allmydata-tahoe-1.10.0\\support\\Lib\\site-packages\\twisted-12.3.0-py2.7-win-amd64.egg\\twisted\\internet\\defer.py\", line 576, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"c:\\allmydata-tahoe-1.10.0\\src\\allmydata\\immutable\\encode.py\", line 501, in _remove_shareholder
    raise UploadUnhappinessError(msg)
allmydata.interfaces.UploadUnhappinessError: shares could be placed or found on only 0 server(s). We were asked to place shares on at least 1 server(s) such that any 1 of them have enough shares to recover the file.: [Failure instance: Traceback (failure with no frames): <class 'allmydata.util.pipeline.PipelineError'>: <PipelineError error=([Failure instance: Traceback (failure with no frames): <class 'foolscap.tokens.RemoteException'>: <RemoteException around '[CopiedFailure instance: Traceback from remote host -- Traceback (most recent call last):
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/tcp.py\", line 299, in connectionLost
    protocol.connectionLost(reason)
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/web/client.py\", line 204, in connectionLost
    self.factory._disconnectedDeferred.callback(None)
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 382, in callback
    self._startRunCallbacks(result)
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 490, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 577, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File \"/home/customer/LAFS_source/src/allmydata/storage/backends/cloud/cloud_common.py\", line 370, in _retry
    d2 = self._handle_error(f, 1, None, description, operation, *args, **kwargs)
  File \"/home/customer/LAFS_source/src/allmydata/storage/backends/cloud/cloud_common.py\", line 417, in _handle_error
    d = task.deferLater(self._reactor, BACKOFF_SECONDS_BEFORE_RETRY[trynum-1], operation, *args, **kwargs)
exceptions.AttributeError: S3Container instance has no attribute '_reactor'
]'>
])>

comment:25 Changed at 2014-03-22T02:27:43Z by CyberAxe

Here's another error after the last one

:: ---------------------  START of LOG  ------------ 
:: START Date: 20142103	Time: 174729 
:: Backup-Now assuming tahoe: as alias and BackedUp as Target Location in Grid 
:: c:\Python27\ must be in system PATH: see tahoe-lafs Windows wiki for help 
:: Will Backup everything in folder c:\BackMeUp\ 
:: v0040  
:: ---------------------  Start Backup  ------------ 
:: ---------------------  Errors Backup  ----------- 
Traceback (most recent call last):
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\cli.py", line 574, in backup
    rc = tahoe_backup.backup(options)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 325, in backup
    return bu.run()
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 123, in run
    put_child(archives_url, now, new_backup_dircap)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 57, in put_child
    raise HTTPError("Error during put_child", resp)
HTTPError: Error during put_child: 500 Internal Server Error
"Traceback (most recent call last):
Failure: allmydata.mutable.common.NotEnoughServersError: ('Publish ran out of good servers, last failure was: [Failure instance: Traceback (failure with no frames): <class \\'foolscap.tokens.RemoteException\\'>: <RemoteException around \\'[CopiedFailure instance: Traceback from remote host -- Traceback (most recent call last):\\n  File \"/usr/lib/python2.7/dist-packages/foolscap/eventual.py\", line 26, in _turn\\n    cb(*args, **kwargs)\\n  File \"/home/customer/LAFS_source/src/allmydata/util/deferredutil.py\", line 55, in _with_log\\n    op(res)\\n  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 382, in callback\\n    self._startRunCallbacks(result)\\n  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 490, in _startRunCallbacks\\n    self._runCallbacks()\\n--- <exception caught here> ---\\n  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 577, in _runCallbacks\\n    current.result = callback(current.result, *args, **kw)\\n  File \"/home/customer/LAFS_source/src/allmydata/storage/backends/base.py\", line 170, in _update_lease\\n    account.add_or_renew_default_lease(self.storage_index, shnum)\\n  File \"/home/customer/LAFS_source/src/allmydata/storage/account.py\", line 58, in add_or_renew_default_lease\\n    return self.add_or_renew_lease(storage_index, shnum, renewal_time, expiration_time)\\n  File \"/home/customer/LAFS_source/src/allmydata/storage/account.py\", line 63, in add_or_renew_lease\\n    renewal_time, expiration_time)\\n  File \"/home/customer/LAFS_source/src/allmydata/storage/leasedb.py\", line 267, in add_or_renew_leases\\n    raise NonExistentShareError(si_s, shnum)\\nallmydata.storage.leasedb.NonExistentShareError: can\\'t find SI=\\'o2ofk2psfahdhyf5uq3nqdmrwe\\' shnum=0 in \`shares\` table\\n]\\'>\\n]', None)

comment:26 in reply to: ↑ 23 Changed at 2014-03-22T02:39:33Z by CyberAxe

I found that my script wasn't showing me it had moved on to a deep-check and was done.

Replying to CyberAxe:

(3:10:26 PM) CyberAxe: Seems like instead of the 500 error now I'm getting it just stuck over and over, maybe I'm not waitting long enought for the 500 error to generate but I've waited hours. http://sebsauvage.net/paste/?f38e2b33f87e1422#A8iAnduHoydJrE6XejuwYX6CHoRx/iWBxUb+OLx1mvM= I have to terminate the process here's what it says when I do that. (3:12:20 PM) CyberAxe: the last line it's stuck on in this example is a create folder action, when I restart backup it does use that folder, so it is creating it re-using old directory for 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files\9' (3:12:20 PM) CyberAxe: re-using old directory for 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files' (3:12:20 PM) CyberAxe: re-using old directory for 'c:\BackMeUp\WesternColorado_files\images' (3:12:20 PM) CyberAxe: re-using old directory for 'c:\BackMeUp\WesternColorado_files' (3:12:20 PM) CyberAxe: re-using old directory for 'c:\BackMeUp\' <----- where it got stuck.

Last edited at 2014-03-22T17:32:30Z by zooko (previous) (diff)

comment:27 Changed at 2014-03-22T17:27:39Z by CyberAxe

Here's what I'm getting now:

processing 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files\9'
skipping 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files\9\0_0.jpg'..
skipping 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files\9\0_1.jpg'..
skipping 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files\9\1_0.jpg'..
skipping 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files\9\1_1.jpg'..
 re-using old directory for 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files\9'
 re-using old directory for 'c:\BackMeUp\WesternColorado_files\images\WaspWar3_files'
 re-using old directory for 'c:\BackMeUp\WesternColorado_files\images'
 re-using old directory for 'c:\BackMeUp\WesternColorado_files'
 creating directory for 'c:\BackMeUp\'
Traceback (most recent call last):
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\cli.py", line 574, in backup
    rc = tahoe_backup.backup(options)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 325, in backup
    return bu.run()
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 123, in run
    put_child(archives_url, now, new_backup_dircap)
  File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 57, in put_child
    raise HTTPError("Error during put_child", resp)
HTTPError: Error during put_child: 500 Internal Server Error
"Traceback (most recent call last):
Failure: allmydata.mutable.common.NotEnoughServersError: ('Publish ran out of good servers, last failure was: [Failure instance: Traceback (
failure with no frames): <class \\'foolscap.tokens.RemoteException\\'>: <RemoteException around \\'[CopiedFailure instance: Traceback from r
emote host -- Traceback (most recent call last):\\n  File \"/usr/lib/python2.7/dist-packages/foolscap/eventual.py\", line 26, in _turn\\n
 cb(*args, **kwargs)\\n  File \"/home/customer/LAFS_source/src/allmydata/util/deferredutil.py\", line 55, in _with_log\\n    op(res)\\n  Fil
e \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 382, in callback\\n    self
._startRunCallbacks(result)\\n  File \"/usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\
", line 490, in _startRunCallbacks\\n    self._runCallbacks()\\n--- <exception caught here> ---\\n  File \"/usr/local/lib/python2.7/dist-pac
kages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py\", line 577, in _runCallbacks\\n    current.result = callback(current.re
sult, *args, **kw)\\n  File \"/home/customer/LAFS_source/src/allmydata/storage/backends/base.py\", line 170, in _update_lease\\n    account.
add_or_renew_default_lease(self.storage_index, shnum)\\n  File \"/home/customer/LAFS_source/src/allmydata/storage/account.py\", line 58, in
add_or_renew_default_lease\\n    return self.add_or_renew_lease(storage_index, shnum, renewal_time, expiration_time)\\n  File \"/home/custom
er/LAFS_source/src/allmydata/storage/account.py\", line 63, in add_or_renew_lease\\n    renewal_time, expiration_time)\\n  File \"/home/cust
omer/LAFS_source/src/allmydata/storage/leasedb.py\", line 267, in add_or_renew_leases\\n    raise NonExistentShareError(si_s, shnum)\\nallmy
data.storage.leasedb.NonExistentShareError: can\\'t find SI=\\'o2ofk2psfahdhyf5uq3nqdmrwe\\' shnum=0 in \`shares\` table\\n]\\'>\\n]', None)

"

comment:28 in reply to: ↑ 22 Changed at 2014-03-25T03:10:49Z by daira

Replying to CyberAxe:

>   File \"/home/customer/LAFS_source/src/allmydata/storage/backends/cloud/cloud_common.py\", line 370, in _retry
>     d2 = self._handle_error(f, 1, None, description, operation, *args, **kwargs)
>   File \"/home/customer/LAFS_source/src/allmydata/storage/backends/cloud/cloud_common.py\", line 417, in _handle_error
>     d = task.deferLater(self._reactor, BACKOFF_SECONDS_BEFORE_RETRY[trynum-1], operation, *args, **kwargs)
> exceptions.AttributeError: S3Container instance has no attribute '_reactor'

This is a separate bug in the S3 retry logic. Filed as #2206.

Last edited at 2014-03-25T03:11:18Z by daira (previous) (diff)

comment:29 Changed at 2014-04-15T00:49:57Z by daira

Resolution set to fixed
Status changed from assigned to closed

Fixed in https://github.com/tahoe-lafs/tahoe-lafs/commit/ea3111d24a97bf9873964383430a9f2e9ff5eb70 on the 1819-cloud-merge branch.

Note: See TracTickets for help on using tickets.

Download in other formats: