#1253 closed defect (fixed)
http proxying can break webapi and leak capabilities (manifests as JSON decoding error in CLI tests)
| Reported by: | davidsarah | Owned by: | warner |
|---|---|---|---|
| Priority: | critical | Milestone: | 1.8.1 |
| Component: | code-frontend-cli | Version: | 1.8.0 |
| Keywords: | json error http proxy security capleak docs reviewed | Cc: | |
| Launchpad Bug: |
Description
On tahoe-dev, Yonggang Wen and Jimmy Tang reported errors with r4803 (nearly current trunk) in the following tests:
Ln
test_ln_with_nonexistent_alias
test_ln_without_alias
Mv
test_mv_behavior
test_mv_with_nonexistent_alias
test_mv_without_alias
SystemTest
test_filesystem
For example:
[ERROR]: allmydata.test.test_cli.Ln.test_ln_with_nonexistent_alias
Traceback (most recent call last):
[...twisted and runner stack entries snipped...]
File
"/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py",
line 509, in ln
rc = tahoe_mv.mv(options, mode="link")
File
"/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py",
line 32, in mv
nodetype, attrs = simplejson.loads(data)
File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line
384, in loads
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line
402, in decode
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line
420, in raw_decode
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1
column 0 (char 0)
Other tests with errors also go through line 32 of tahoe_mv.py.
It's not clear yet whether simplejson is failing to decode valid JSON, or whether the gateway is producing JSON that is not strictly valid.
Attachments (3)
Change History (25)
comment:1 follow-up: ↓ 3 Changed at 2010-11-09T22:13:04Z by jcftang
comment:2 follow-up: ↓ 4 Changed at 2010-11-09T23:23:34Z by davidsarah
- Component changed from code-frontend-cli to code-frontend
- Keywords http proxy squid added
- Summary changed from JSON decoding error in CLI tests on an OS X 10.6 system to http proxying can break webapi (manifests as JSON decoding error in CLI tests)
OK, this seems to be due to a squid proxy intercepting webapi requests. Jimmy Tang's comment on tahoe-dev supports that conclusion:
After making the change you suggested I noticed some of the tests were trying to use my proxy when running some of the tests. Interestingly when I unset the http_proxy and HTTP_PROXY environment variables the tests that failed before now passes.
Presumably this would also break actual use of the webapi, not just tests.
In theory, requests to 127.0.0.1 should either not be intercepted or should be passed through transparently, but that's not happening for this particular proxy configuration. The proxy is returning the following HTML error document, which is obviously not valid JSON:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The requested URL could not be retrieved</TITLE>
<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>
</HEAD><BODY>
<H1>ERROR</H1>
<H2>The requested URL could not be retrieved</H2>
<HR noshade size="1px">
<P>
While trying to retrieve the URL:
<A HREF="http://127.0.0.1:56496/uri/URI%3ADIR2%3Ab5hqzkgxe2v6lugxnnfs5p5h3q%3A24xpssxc6jpouua2ld3opk4urytcwyql6jhxnil3bqd2ffrpigiq/file1?">http://127.0.0.1:56496/uri/URI%3ADIR2%3Ab5hqzkgxe2v6lugxnnfs5p5h3q%3A24xpssxc6jpouua2ld3opk4urytcwyql6jhxnil3bqd2ffrpigiq/file1?</A>
<P>
The following error was encountered:
<UL>
<LI>
<STRONG>
Access Denied.
</STRONG>
<P>
Access control configuration prevents your request from
being allowed at this time. Please contact your service provider if
you feel this is incorrect.
</UL>
<P>Your cache administrator is <A HREF="mailto:root">root</A>.
<BR clear="all">
<HR noshade size="1px">
<ADDRESS>
Generated Tue, 09 Nov 2010 21:38:22 GMT by gateway.tchpc.tcd.ie (squid/2.6.STABLE21)
</ADDRESS>
</BODY></HTML>
Note that any response that starts with a '<' (or a BOM followed by '<') is likely to be HTML coming from something other than the intended gateway; we should give a better error report in that case.
comment:3 in reply to: ↑ 1 Changed at 2010-11-09T23:38:52Z by davidsarah
Replying to jcftang:
===============================================================================
[FAIL]: allmydata.test.test_client.Basic.test_reserved_4
Traceback (most recent call last):
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/test/test_client.py", line 122, in test_reserved_4
78*1000*1000*1000)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/test/common_util.py", line 32, in failUnlessReallyEqual
self.failUnlessEqual(type(a), type(b), msg="a :: %r, b :: %r, %r" % (a, b, msg))
twisted.trial.unittest.FailTest: a :: 78000000000, b :: 78000000000L, None
not equal:
a = <type 'int'>
b = <type 'long'>
}}}
This failure (which has nothing to do with the http proxy issue) should have been fixed by edec9c986917e8f4. jcftang: please check that with current trunk.
comment:4 in reply to: ↑ 2 ; follow-up: ↓ 5 Changed at 2010-11-09T23:58:47Z by davidsarah
- Owner set to jcftang
Replying to davidsarah:
OK, this seems to be due to a squid proxy intercepting webapi requests. Jimmy Tang's comment on tahoe-dev supports that conclusion:
After making the change you suggested I noticed some of the tests were trying to use my proxy when running some of the tests. Interestingly when I unset the http_proxy and HTTP_PROXY environment variables the tests that failed before now passes.
Please try setting the http_proxy and HTTP_PROXY variables as they were before, and the no_proxy variable to "127.0.0.1:56496".
comment:5 in reply to: ↑ 4 Changed at 2010-11-10T00:00:57Z by davidsarah
Replying to davidsarah:
Please try setting the http_proxy and HTTP_PROXY variables as they were before, and the no_proxy variable to "127.0.0.1:56496".
Oh, that probably won't work because the port is dynamic. Try setting no_proxy just to "127.0.0.1" instead.
comment:6 Changed at 2010-11-10T00:14:52Z by warner
hm, perhaps we should add an early test (i.e. alphabetically first) which just tries a simple HTTP request to self, to catch things like HTTP_PROXY? The error message on that test could look at os.environ["HTTP_PROXY"] and mention that it might be interfering with the self-tests, and remind users to configure their proxies to allow 127.0.0.1 to connect directly.
(Foolscap has a test__versions.py that runs before everything else, so that its output is easier to find.. we could add a test__http_proxy.py to Tahoe and maybe even os.abort() the whole test run if it fails, to avoid later confusion)
comment:7 Changed at 2010-11-10T00:29:18Z by jcftang
in reply to comment:5, i tried setting the no_proxy variable and leaving my http_proxy (and friends) variables alone, I now get errors such as...
Ln
test_ln_with_nonexistent_alias ... "<html><head><title>Page Not Found</title></head><body>Sorry, but I couldn't find the object you requested.</body></html>"
[ERROR]
test_ln_without_alias ... "<html><head><title>Page Not Found</title></head><body>Sorry, but I couldn't find the object you requested.</body></html>"
[ERROR]
Manifest
test_manifest_with_nonexistent_alias ... [OK]
test_manifest_without_alias ... [OK]
Mkdir
test_mkdir ... [OK]
test_mkdir_unicode ... [SKIPPED]
test_mkdir_with_nonexistent_alias ... [OK]
Mv
test_mv_behavior ... "<html><head><title>Page Not Found</title></head><body>Sorry, but I couldn't find the object you requested.</body></html>"
[ERROR]
test_mv_with_nonexistent_alias ... "<html><head><title>Page Not Found</title></head><body>Sorry, but I couldn't find the object you requested.</body></html>"
[ERROR]
test_mv_without_alias ... "<html><head><title>Page Not Found</title></head><body>Sorry, but I couldn't find the object you requested.</body></html>"
[ERROR]
Put
allmydata.test.test_system
SystemTest
test_filesystem ... "<html><head><title>Page Not Found</title></head><body>Sorry, but I couldn't find the object you requested.</body></html>"
[ERROR]
the end of the tests... just to be sure im pasting the correct error messages...
===============================================================================
[ERROR]: allmydata.test.test_cli.Ln.test_ln_with_nonexistent_alias
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext
return func(*args,**kw)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner
rc = cli.dispatch[command](so)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 509, in ln
rc = tahoe_mv.mv(options, mode="link")
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv
print repr(data); nodetype, attrs = simplejson.loads(data)
File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
===============================================================================
[ERROR]: allmydata.test.test_cli.Ln.test_ln_without_alias
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext
return func(*args,**kw)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner
rc = cli.dispatch[command](so)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 509, in ln
rc = tahoe_mv.mv(options, mode="link")
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv
print repr(data); nodetype, attrs = simplejson.loads(data)
File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
===============================================================================
[ERROR]: allmydata.test.test_cli.Mv.test_mv_behavior
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext
return func(*args,**kw)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner
rc = cli.dispatch[command](so)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 504, in mv
rc = tahoe_mv.mv(options, mode="move")
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv
print repr(data); nodetype, attrs = simplejson.loads(data)
File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
===============================================================================
[ERROR]: allmydata.test.test_cli.Mv.test_mv_with_nonexistent_alias
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext
return func(*args,**kw)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner
rc = cli.dispatch[command](so)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 504, in mv
rc = tahoe_mv.mv(options, mode="move")
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv
print repr(data); nodetype, attrs = simplejson.loads(data)
File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
===============================================================================
[ERROR]: allmydata.test.test_cli.Mv.test_mv_without_alias
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext
return func(*args,**kw)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner
rc = cli.dispatch[command](so)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 504, in mv
rc = tahoe_mv.mv(options, mode="move")
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv
print repr(data); nodetype, attrs = simplejson.loads(data)
File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
===============================================================================
[ERROR]: allmydata.test.test_system.SystemTest.test_filesystem
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext
return func(*args,**kw)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner
rc = cli.dispatch[command](so)
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 504, in mv
rc = tahoe_mv.mv(options, mode="move")
File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv
print repr(data); nodetype, attrs = simplejson.loads(data)
File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode
File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
-------------------------------------------------------------------------------
Ran 828 tests in 366.156s
FAILED (skips=15, expectedFailures=3, failures=1, errors=6, successes=803)
1|jtang@x00:...llmydata-tahoe-1.8.0-r4803 $
in response to comment:3 I will need to download the trunk from the trac interface when I'm in work tomorrow morning to test again. I've so far been testing the near trunk branch from the git repo.
comment:8 Changed at 2010-11-10T00:34:18Z by davidsarah
Hmm, I see the problem. Most of our code uses httplib (which does not make use of the http_proxy or HTTP_PROXY variables) to establish HTTP connections, but tahoe_mv.py uses urllib.urllopen here. That's why only the tests of tahoe mv are failing.
(There is another use of urllib.urlopen in src/allmydata/test/check_load.py, but that's not run by setup.py test.)
comment:9 Changed at 2010-11-10T00:57:31Z by davidsarah
- Component changed from code-frontend to code-frontend-cli
- Keywords security capleak added; squid removed
- Owner changed from jcftang to davidsarah
- Priority changed from major to critical
- Status changed from new to assigned
This could cause capabilities (specifically, the source arguments to tahoe mv or tahoe ln) to be leaked to the HTTP proxy server. The proxy server is not in our reliance diagram and therefore not supposed to be trusted.
comment:10 Changed at 2010-11-10T01:12:06Z by davidsarah
- Keywords docs news-needed test-needed added; test removed
- Summary changed from http proxying can break webapi (manifests as JSON decoding error in CLI tests) to http proxying can break webapi and leak capabilities (manifests as JSON decoding error in CLI tests)
Changed at 2010-11-10T01:38:13Z by davidsarah
tahoe_mv.py: use do_http, not urllib.openurl, in order to avoid connecting to the webapi server via a proxy. refs #1253
Changed at 2010-11-10T04:05:56Z by davidsarah
test_system.py: test for 'tahoe mv' with the http_proxy and HTTP_PROXY variables set. refs #1253
comment:11 Changed at 2010-11-10T04:06:46Z by davidsarah
- Keywords review-needed added; test-needed removed
- Owner changed from davidsarah to warner
- Status changed from assigned to new
comment:12 Changed at 2010-11-10T08:43:59Z by zooko
- Keywords reviewed added; review-needed removed
- Owner changed from warner to zooko
- Status changed from new to assigned
I reviewed the patches and they look good. We still need a source:NEWS entry for this ticket. Here, please review this patch that I'm about to attach that adds a NEWS entry.
Changed at 2010-11-10T08:45:18Z by zooko
comment:13 Changed at 2010-11-10T08:45:33Z by zooko
- Owner changed from zooko to warner
- Status changed from assigned to new
comment:14 Changed at 2010-11-10T19:13:27Z by davidsarah
- Keywords news-needed removed
Reviewed the news patch. Looks good, but please make the changes to configuration.rst and hashsig.py into separate patches.
comment:15 Changed at 2010-11-11T06:25:43Z by zooko@…
In 531443792e3334a7:
comment:16 Changed at 2010-11-11T06:25:47Z by david-sarah@…
In 01a53650510a9a4e:
comment:17 Changed at 2010-11-11T06:25:47Z by david-sarah@…
In cb777ad14f12a249:
comment:18 Changed at 2010-11-11T09:12:26Z by jcftang
I just did a checkout of the darcs repo, and re-ran the tests in my environment that failed the last time. The tests passes now.
comment:19 Changed at 2010-11-11T13:48:12Z by zooko
- Resolution set to fixed
- Status changed from new to closed
Fixed in 01a53650510a9a4e, cb777ad14f12a249. Thank you!
comment:20 Changed at 2010-11-13T08:01:29Z by zooko
- Resolution fixed deleted
- Status changed from closed to reopened
There is now a test failure on zomp (Mac OS 10.6) which looks like it might be caused by these patches:
[FAIL]
Traceback (most recent call last):
File "/Users/tahoebuildslave/Zooko zomp Mac-amd64 10.6 py2.6/build/src/allmydata/test/test_system.py", line 1609, in _check_mv_with_http_proxy
self.failUnlessEqual(rc_or_sig, 0, str(res))
twisted.trial.unittest.FailTest: ('', 'Traceback (most recent call last):\n File "/Users/tahoebuildslave/Zooko zomp Mac-amd64 10.6 py2.6/build/support/bin/tahoe", line 9, in <module>\n load_entry_point(\'allmydata-tahoe==1.8.0-r4818\', \'console_scripts\', \'tahoe\')()\n File "/Library/Python/2.6/site-packages/allmydata/scripts/runner.py", line 114, in run\n rc = runner(sys.argv[1:], install_node_control=install_node_control)\n File "/Library/Python/2.6/site-packages/allmydata/scripts/runner.py", line 100, in runner\n rc = cli.dispatch[command](so)\n File "/Library/Python/2.6/site-packages/allmydata/scripts/cli.py", line 504, in mv\n rc = tahoe_mv.mv(options, mode="move")\n File "/Library/Python/2.6/site-packages/allmydata/scripts/tahoe_mv.py", line 31, in mv\n data = urllib.urlopen(from_url + "?t=json").read()\n File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 87, in urlopen\n return opener.open(url)\n File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 203, in open\n return getattr(self, name)(url)\n File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 338, in open_http\n h.endheaders()\n File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 868, in endheaders\n self._send_output()\n File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 740, in _send_output\n self.send(msg)\n File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 699, in send\n self.connect()\n File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 683, in connect\n self.timeout)\n File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/socket.py", line 512, in create_connection\n raise error, msg\nIOError: [Errno socket error] [Errno 60] Operation timed out\n', 1)
not equal:
a = 1
b = 0
comment:21 follow-up: ↓ 22 Changed at 2010-11-14T03:33:46Z by davidsarah
- Resolution set to fixed
- Status changed from reopened to closed
src/allmydata/scripts/tahoe_mv.py@4818#L31 is not "data = urllib.urlopen(from_url + "?t=json").read()". That's what it was before the fix.
Somehow the code being tested has the test patch 01a53650510a9a4e, but not the fix patch cb777ad14f12a249.
Oh, it is testing "/Library/Python?/2.6/site-packages/allmydata/scripts/tahoe_mv.py". Wrong! This looks like the issue that we fixed for the test-from-egg and test-from-prefixdir steps in #1137.
Anyway, this failure is not evidence of this ticket being unfixed, so I'm closing it and opening a new ticket for the wrong-code problem.
modified /Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py by changing line 32 to:
I then did a python setup.py build and re-ran the tests as instructed. I've copied and pasted anything that failed with an [ERROR] below
Ln test_ln_with_nonexistent_alias ... '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">\n<TITLE>ERROR: The requested URL could not be retrieved</TITLE>\n<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>\n</HEAD><BODY>\n<H1>ERROR</H1>\n<H2>The requested URL could not be retrieved</H2>\n<HR noshade size="1px">\n<P>\nWhile trying to retrieve the URL:\n<A HREF="http://127.0.0.1:56478/uri/URI%3ADIR2%3Ax4hxl27neryqpr7d6djcjfnlze%3A5hpb37l2mwfnzqrrgdbgmj6xaw3mlb225tifft7mz65a4dluvfia/from?">http://127.0.0.1:56478/uri/URI%3ADIR2%3Ax4hxl27neryqpr7d6djcjfnlze%3A5hpb37l2mwfnzqrrgdbgmj6xaw3mlb225tifft7mz65a4dluvfia/from?</A>\n<P>\nThe following error was encountered:\n<UL>\n<LI>\n<STRONG>\nAccess Denied.\n</STRONG>\n<P>\nAccess control configuration prevents your request from\nbeing allowed at this time. Please contact your service provider if\nyou feel this is incorrect.\n</UL>\n<P>Your cache administrator is <A HREF="mailto:root">root</A>. \n\n\n<BR clear="all">\n<HR noshade size="1px">\n<ADDRESS>\nGenerated Tue, 09 Nov 2010 21:38:18 GMT by gateway.tchpc.tcd.ie (squid/2.6.STABLE21)\n</ADDRESS>\n</BODY></HTML>\n' [ERROR] test_ln_without_alias ... '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">\n<TITLE>ERROR: The requested URL could not be retrieved</TITLE>\n<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>\n</HEAD><BODY>\n<H1>ERROR</H1>\n<H2>The requested URL could not be retrieved</H2>\n<HR noshade size="1px">\n<P>\nWhile trying to retrieve the URL:\n<A HREF="http://127.0.0.1:56482/uri/URI%3ADIR2%3Aobass4lyms5yfudex32l4cytra%3A46oia2riunpahps3bspwofwlid64gqihfygngfulh33cuclqyiya/from?">http://127.0.0.1:56482/uri/URI%3ADIR2%3Aobass4lyms5yfudex32l4cytra%3A46oia2riunpahps3bspwofwlid64gqihfygngfulh33cuclqyiya/from?</A>\n<P>\nThe following error was encountered:\n<UL>\n<LI>\n<STRONG>\nAccess Denied.\n</STRONG>\n<P>\nAccess control configuration prevents your request from\nbeing allowed at this time. Please contact your service provider if\nyou feel this is incorrect.\n</UL>\n<P>Your cache administrator is <A HREF="mailto:root">root</A>. \n\n\n<BR clear="all">\n<HR noshade size="1px">\n<ADDRESS>\nGenerated Tue, 09 Nov 2010 21:38:19 GMT by gateway.tchpc.tcd.ie (squid/2.6.STABLE21)\n</ADDRESS>\n</BODY></HTML>\n' [ERROR]Mv test_mv_behavior ... '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">\n<TITLE>ERROR: The requested URL could not be retrieved</TITLE>\n<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>\n</HEAD><BODY>\n<H1>ERROR</H1>\n<H2>The requested URL could not be retrieved</H2>\n<HR noshade size="1px">\n<P>\nWhile trying to retrieve the URL:\n<A HREF="http://127.0.0.1:56496/uri/URI%3ADIR2%3Ab5hqzkgxe2v6lugxnnfs5p5h3q%3A24xpssxc6jpouua2ld3opk4urytcwyql6jhxnil3bqd2ffrpigiq/file1?">http://127.0.0.1:56496/uri/URI%3ADIR2%3Ab5hqzkgxe2v6lugxnnfs5p5h3q%3A24xpssxc6jpouua2ld3opk4urytcwyql6jhxnil3bqd2ffrpigiq/file1?</A>\n<P>\nThe following error was encountered:\n<UL>\n<LI>\n<STRONG>\nAccess Denied.\n</STRONG>\n<P>\nAccess control configuration prevents your request from\nbeing allowed at this time. Please contact your service provider if\nyou feel this is incorrect.\n</UL>\n<P>Your cache administrator is <A HREF="mailto:root">root</A>. \n\n\n<BR clear="all">\n<HR noshade size="1px">\n<ADDRESS>\nGenerated Tue, 09 Nov 2010 21:38:22 GMT by gateway.tchpc.tcd.ie (squid/2.6.STABLE21)\n</ADDRESS>\n</BODY></HTML>\n' [ERROR] test_mv_with_nonexistent_alias ... '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">\n<TITLE>ERROR: The requested URL could not be retrieved</TITLE>\n<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>\n</HEAD><BODY>\n<H1>ERROR</H1>\n<H2>The requested URL could not be retrieved</H2>\n<HR noshade size="1px">\n<P>\nWhile trying to retrieve the URL:\n<A HREF="http://127.0.0.1:56505/uri/URI%3ADIR2%3Aarnggtsue3l3v36xy7j6cgkbum%3Av7ymzrpg7lnlrmohfytg3vwa6zpzvwtomcdqkbqukx4iryrya7tq/afile?">http://127.0.0.1:56505/uri/URI%3ADIR2%3Aarnggtsue3l3v36xy7j6cgkbum%3Av7ymzrpg7lnlrmohfytg3vwa6zpzvwtomcdqkbqukx4iryrya7tq/afile?</A>\n<P>\nThe following error was encountered:\n<UL>\n<LI>\n<STRONG>\nAccess Denied.\n</STRONG>\n<P>\nAccess control configuration prevents your request from\nbeing allowed at this time. Please contact your service provider if\nyou feel this is incorrect.\n</UL>\n<P>Your cache administrator is <A HREF="mailto:root">root</A>. \n\n\n<BR clear="all">\n<HR noshade size="1px">\n<ADDRESS>\nGenerated Tue, 09 Nov 2010 21:38:22 GMT by gateway.tchpc.tcd.ie (squid/2.6.STABLE21)\n</ADDRESS>\n</BODY></HTML>\n' [ERROR] test_mv_without_alias ... '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">\n<TITLE>ERROR: The requested URL could not be retrieved</TITLE>\n<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>\n</HEAD><BODY>\n<H1>ERROR</H1>\n<H2>The requested URL could not be retrieved</H2>\n<HR noshade size="1px">\n<P>\nWhile trying to retrieve the URL:\n<A HREF="http://127.0.0.1:56509/uri/URI%3ADIR2%3Ax3q34zigp4pnqjyr4oeg5zxqiq%3Akbfrukaeqld4dsnskbzf7oqk772gjel2siztmiio6h4l37jg3ara/afile?">http://127.0.0.1:56509/uri/URI%3ADIR2%3Ax3q34zigp4pnqjyr4oeg5zxqiq%3Akbfrukaeqld4dsnskbzf7oqk772gjel2siztmiio6h4l37jg3ara/afile?</A>\n<P>\nThe following error was encountered:\n<UL>\n<LI>\n<STRONG>\nAccess Denied.\n</STRONG>\n<P>\nAccess control configuration prevents your request from\nbeing allowed at this time. Please contact your service provider if\nyou feel this is incorrect.\n</UL>\n<P>Your cache administrator is <A HREF="mailto:root">root</A>. \n\n\n<BR clear="all">\n<HR noshade size="1px">\n<ADDRESS>\nGenerated Tue, 09 Nov 2010 21:38:23 GMT by gateway.tchpc.tcd.ie (squid/2.6.STABLE21)\n</ADDRESS>\n</BODY></HTML>\n' [ERROR]allmydata.test.test_system SystemTest test_filesystem ... '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">\n<TITLE>ERROR: The requested URL could not be retrieved</TITLE>\n<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>\n</HEAD><BODY>\n<H1>ERROR</H1>\n<H2>The requested URL could not be retrieved</H2>\n<HR noshade size="1px">\n<P>\nWhile trying to retrieve the URL:\n<A HREF="http://127.0.0.1:57270/uri/URI%3ADIR2%3Atrewrecp3fyucy3noidnya4tl4%3Ah5mmrlbxf2r2ywcuvjrp46gq65nqmvjrcr7lt2rbrkjfnvt6hj4a/tahoe-file-stdin?">http://127.0.0.1:57270/uri/URI%3ADIR2%3Atrewrecp3fyucy3noidnya4tl4%3Ah5mmrlbxf2r2ywcuvjrp46gq65nqmvjrcr7lt2rbrkjfnvt6hj4a/tahoe-file-stdin?</A>\n<P>\nThe following error was encountered:\n<UL>\n<LI>\n<STRONG>\nAccess Denied.\n</STRONG>\n<P>\nAccess control configuration prevents your request from\nbeing allowed at this time. Please contact your service provider if\nyou feel this is incorrect.\n</UL>\n<P>Your cache administrator is <A HREF="mailto:root">root</A>. \n\n\n<BR clear="all">\n<HR noshade size="1px">\n<ADDRESS>\nGenerated Tue, 09 Nov 2010 21:41:33 GMT by gateway.tchpc.tcd.ie (squid/2.6.STABLE21)\n</ADDRESS>\n</BODY></HTML>\n' [ERROR]=============================================================================== [FAIL]: allmydata.test.test_client.Basic.test_reserved_4 Traceback (most recent call last): File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/test/test_client.py", line 122, in test_reserved_4 78*1000*1000*1000) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/test/common_util.py", line 32, in failUnlessReallyEqual self.failUnlessEqual(type(a), type(b), msg="a :: %r, b :: %r, %r" % (a, b, msg)) twisted.trial.unittest.FailTest: a :: 78000000000, b :: 78000000000L, None not equal: a = <type 'int'> b = <type 'long'> =============================================================================== [ERROR]: allmydata.test.test_cli.Ln.test_ln_with_nonexistent_alias Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, *args, **kwargs) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner rc = cli.dispatch[command](so) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 509, in ln rc = tahoe_mv.mv(options, mode="link") File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv print repr(data); nodetype, attrs = simplejson.loads(data) File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) =============================================================================== [ERROR]: allmydata.test.test_cli.Ln.test_ln_without_alias Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, *args, **kwargs) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner rc = cli.dispatch[command](so) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 509, in ln rc = tahoe_mv.mv(options, mode="link") File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv print repr(data); nodetype, attrs = simplejson.loads(data) File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) =============================================================================== [ERROR]: allmydata.test.test_cli.Mv.test_mv_behavior Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, *args, **kwargs) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner rc = cli.dispatch[command](so) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 504, in mv rc = tahoe_mv.mv(options, mode="move") File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv print repr(data); nodetype, attrs = simplejson.loads(data) File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) =============================================================================== [ERROR]: allmydata.test.test_cli.Mv.test_mv_with_nonexistent_alias Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, *args, **kwargs) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner rc = cli.dispatch[command](so) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 504, in mv rc = tahoe_mv.mv(options, mode="move") File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv print repr(data); nodetype, attrs = simplejson.loads(data) File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) =============================================================================== [ERROR]: allmydata.test.test_cli.Mv.test_mv_without_alias Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, *args, **kwargs) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner rc = cli.dispatch[command](so) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 504, in mv rc = tahoe_mv.mv(options, mode="move") File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv print repr(data); nodetype, attrs = simplejson.loads(data) File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0) =============================================================================== [ERROR]: allmydata.test.test_system.SystemTest.test_filesystem Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, *args, **kwargs) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/runner.py", line 100, in runner rc = cli.dispatch[command](so) File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/cli.py", line 504, in mv rc = tahoe_mv.mv(options, mode="move") File "/Users/jtang/sandbox/allmydata-tahoe-1.8.0-r4803/src/allmydata/scripts/tahoe_mv.py", line 33, in mv print repr(data); nodetype, attrs = simplejson.loads(data) File "build/bdist.macosx-10.6-universal/egg/simplejson/__init__.py", line 384, in loads File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 402, in decode File "build/bdist.macosx-10.6-universal/egg/simplejson/decoder.py", line 420, in raw_decode simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)