Opened at 2014-06-30T16:49:44Z
Last modified at 2014-07-01T15:49:46Z
#2254 new defect
Can't backup still suffering from allmydata.interfaces.UploadUnhappinessError
| Reported by: | CyberAxe | Owned by: | daira |
|---|---|---|---|
| Priority: | normal | Milestone: | undecided |
| Component: | unknown | Version: | 1.10.0 |
| Keywords: | unhappy | Cc: | |
| Launchpad Bug: |
Description (last modified by CyberAxe)
C:\Users\Jeremy>backup-now
C:\Users\Jeremy>ECHO OFF
Traceback (most recent call last):
File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 156, in run
rc = runner(sys.argv[1:], install_node_control=install_node_control)
File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\runner.py", line 141, in runner
rc = cli.dispatch[command](so)
File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\cli.py", line 574, in backup
rc = tahoe_backup.backup(options)
File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 325, in backup
return bu.run()
File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 118, in run
new_backup_dircap = self.process(options.from_dir)
File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 211, in process
newdircap = mkdir(create_contents, self.options)
File "c:\allmydata-tahoe-1.10.0\src\allmydata\scripts\tahoe_backup.py", line 47, in mkdir
raise HTTPError("Error during mkdir", resp)
HTTPError: Error during mkdir: 500 Internal Server Error
Traceback (most recent call last):
File "C:\allmydata-tahoe-1.10.0\support\Lib\site-packages\foolscap-0.6.4-py2.7.egg\foolscap\call.py", line 677, in _done
self.request.complete(res)
File "C:\allmydata-tahoe-1.10.0\support\Lib\site-packages\foolscap-0.6.4-py2.7.egg\foolscap\call.py", line 60, in complete
self.deferred.callback(res)
File "C:\allmydata-tahoe-1.10.0\support\Lib\site-packages\twisted-12.3.0-py2.7-win-amd64.egg\twisted\internet\defer.py", line 381, in call
back
self._startRunCallbacks(result)
File "C:\allmydata-tahoe-1.10.0\support\Lib\site-packages\twisted-12.3.0-py2.7-win-amd64.egg\twisted\internet\defer.py", line 489, in _sta
rtRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "C:\allmydata-tahoe-1.10.0\support\Lib\site-packages\twisted-12.3.0-py2.7-win-amd64.egg\twisted\internet\defer.py", line 576, in _run
Callbacks
current.result = callback(current.result, *args, **kw)
File "c:\allmydata-tahoe-1.10.0\src\allmydata\immutable\upload.py", line 604, in _got_response
return self._loop()
File "c:\allmydata-tahoe-1.10.0\src\allmydata\immutable\upload.py", line 516, in _loop
return self._failed(msg)
File "c:\allmydata-tahoe-1.10.0\src\allmydata\immutable\upload.py", line 617, in _failed
raise UploadUnhappinessError(msg)
allmydata.interfaces.UploadUnhappinessError: server selection failed for <Tahoe2ServerSelector for upload 66ww6>: shares could be placed or
found on only 0 server(s). We were asked to place shares on at least 1 server(s) such that any 1 of them have enough shares to recover the f
ile. (placed 0 shares out of 1 total (1 homeless), want to place shares on at least 1 servers such that any 1 of them have enough shares to
recover the file, sent 1 queries to 1 servers, 0 queries placed some shares, 1 placed none (of which 1 placed none due to the server being f
ull and 0 placed none due to an error))
Change History (5)
comment:1 Changed at 2014-06-30T16:50:17Z by CyberAxe
- Description modified (diff)
comment:2 Changed at 2014-06-30T16:54:44Z by CyberAxe
comment:3 follow-up: ↓ 4 Changed at 2014-06-30T19:47:59Z by zooko
comment:4 in reply to: ↑ 3 ; follow-up: ↓ 5 Changed at 2014-06-30T22:54:01Z by Zancas
Replying to zooko:
This is probably related to #2251 -- it is happening on the same server as #2251 is and both error messages have to do with the server's filesystem being full.
As mentioned in #2251 the SSEC2 filesystem was _not_ full after June 28.
Here's the output of df:
customer@ip-10-185-214-61:~/storageserver/logs$ date Mon Jun 30 22:57:08 UTC 2014 customer@ip-10-185-214-61:~/storageserver/logs$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 8256952 1528136 6309388 20% / udev 299036 8 299028 1% /dev tmpfs 60948 160 60788 1% /run none 5120 0 5120 0% /run/lock none 304720 0 304720 0% /run/shm
comment:5 in reply to: ↑ 4 Changed at 2014-07-01T15:49:46Z by zooko
Replying to Zancas:
Replying to zooko:
This is probably related to #2251 -- it is happening on the same server as #2251 is and both error messages have to do with the server's filesystem being full.
As mentioned in #2251 the SSEC2 filesystem was _not_ full after June 28.
Okay, then I suspect this issue -- #2254 -- may be due to corruption caused by the filesystem-full condition but not cured by the freeing-up of filesystem space. We need to look for incident report files and other evidence from the affected storage server.
Note: See
TracTickets for help on using
tickets.
Pastebin messed this up so I just posted it here. This is what http://127.0.0.1:3456/status/ shows