DeprecationWarning about passing non-bytes header values

[daira]

​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Marcus%20Cygwin%20WinXP/builds/26/steps/test/logs/stdio

/home/marcus/bbtahoe/Marcus_Cygwin_WinXP/build/support/lib/python2.6/site-packages/Twisted-13.0.0-py2.6-cygwin-1.7.9-i686.egg/twisted/web/server.py:220:
DeprecationWarning: Passing non-bytes header values is deprecated since Twisted 12.3. Pass only bytes instead.

I don't know why this is warning only on the "Marcus Cygwin WinXP" builder.

[daira]

Ah, it is also happening on the Python 2.6 Travis build: ​https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/36701619

The common factor is Python 2.6.

[warner]

This is low-priority, but it'd be nice to fix. It appears that something is calling http.Request.write with a unicode string, but we need a larger stack trace to figure out what's happening.

[warner]

A minimal STR is to test test_web.Web.test_GET_DIRURL_json under python2.6 .

It turns out that the twisted.web code is expecting the Content-Length header to contain bytes (like all other headers, who knew?). Three Tahoe code paths (in web/common.py and web/filenode.py) fill it with an integer, which takes advantage of a ("%s" % value) fallback.

A DeprecationWarning to complain about non-bytes was added in Twisted-12.3, probably to guard against unicode header values, but in this case it's triggering on our IntType. Twisted-12.3 is also when twisted.python.compat.intToBytes(n) was added to make this easier, but it isn't present in 11.0-12.2, which is what we use on windows. So we'll need our own intToBytes function.

[warner]

Oh, and category=DeprecationWarning is globally disabled in python2.7, because(?) they assumed it would only be used for features implemented by python itself, and "2.7 is the last one". I think that's why this doesn't appear in py2.7 .

Sigh, that seems like a dangerous misfeature to me. Should Twisted (and other projects) use a subclass or alternative to DeprecationWarning?

[warner]

After fixing that, I found one other instance, when various web requests use nevow.rend.Page's feature of self.addSlash==True. This code path causes setHeader("location", url) to be called with a nevow.url.URL instance, instead of a string. There's no good way for us to fix this from tahoe.. the fix really needs to be in Nevow itself. Filed as ​https://github.com/twisted/nevow/issues/52 .

[warner]

Ok, PR in ​https://github.com/tahoe-lafs/tahoe-lafs/pull/137 , ready for review. This will fix the immediate problem, but there will probably be others (including ones we can't fix because they're in Nevow or other libraries). I'll work on setting up a buildslave that will run tests with PYTHONWARNINGS=default::DeprecationWarning so we can at least see where they're happening.

[warner]

#2366 created to remind me about the warnings-scanning buildbot tool

[daira]

[6f74282688a871c5ef8cb0d472b2050edc8e6102/trunk] has the patch but it does not seem to have fixed the warnings: ​https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/47869702

[zooko]

removing review-needed tag (which by the way was incorrectly spelled needs-review)

The next step is: figure out why the patch didn't fix the issue?

[warner]

I suspect the remaining warnings are resulting from Nevow's addSlash=True problem. I don't think we can fix that within Tahoe, but I could be wrong.

I think we've gotten as much out of this ticket as we can for 1.10.1 . I propose we bump it to 1.11 at this week's triage.

[daira]

If we can't fix it, should we suppress the warning? We have done so for previous warnings that could only be fixed in dependencies.

[warner]

let's make sure that this has been reported upstream to Nevow, if possible

[warner]

Nevow bug filed: ​https://github.com/twisted/nevow/issues/52 . (oh, as I reported earlier in comment:6, nevermind).

Buildbot is now scanning for deprecation warnings, ​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Ubuntu%20trusty%2014.04/builds/45/steps/deprecations/logs/warnings has a list. This is a simple regexp scan, so some of the lines are coming from the tests that fail (due to stderr not being as quiet as it's supposed to be):

/home/bb-tahoe/bb-tahoe/Ubuntu_trusty_14_04/build/support/lib/python2.7/site-packages/Nevow-0.11.1-py2.7.egg/nevow/appserver.py:129: DeprecationWarning: twisted.web.http.Request.received_headers was deprecated in Twisted 13.2.0: Please use twisted.web.http.Request.requestHeaders instead.
/usr/lib/python2.7/dist-packages/twisted/web/server.py:217: DeprecationWarning: Passing non-bytes header values is deprecated since Twisted 12.3. Pass only bytes instead.
a = '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated\n  import sets as _sets\n'
twisted.trial.unittest.FailTest: ('allmydata-tahoe: 1.10.0.post287 [master: 681eb4d12e20d8818ee02baab7b55230a706e76d]\nfoolscap: 0.6.4\npycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958\nzfec: 1.4.5\nTwisted: 13.2.0\nNevow: 0.11.1\nzope.interface: unknown\npython: 2.7.6\nplatform: Linux-Ubuntu_14.04-x86_64-32bit_ELF\npyOpenSSL: 0.13\nsimplejson: 3.3.1\npycrypto: 2.6.1\npyasn1: 0.1.7\nmock: 1.0.1\nsetuptools: 0.6c16dev5\nservice-identity: 14.0.0\ncharacteristic: 14.3.0\npyasn1-modules: 0.0.5\n\n', '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated\n  import sets as _sets\n', 0)
twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated

So there are three real warnings:

• nevow/appserver.py:129: "twisted.web.http.Request.received_headers was deprecated in Twisted 13.2.0"
• twisted/web/server.py:217: "Passing non-bytes header values is deprecated since Twisted 12.3"
• twisted/spread/jelly.py:92: "the sets module is deprecated; import sets as _sets"

The first is filed as ​https://github.com/twisted/nevow/issues/53 , and the second as ​https://github.com/twisted/nevow/issues/52 . I don't know if the third is filed anywhere.

[warner]

All the bugs (that might ever get fixed) are filed, a separate ticket will track the creation of a buildbot to look for more. I think this ticket is done.