Running unit tests throws DeprecationWarnings

[imhavoc]

bigpig@~ $ tahoe restart
STOPPING /home/bigpig/.tahoe
process 11011 is dead
STARTING /home/bigpig/.tahoe
/usr/lib/pymodules/python2.6/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated
  import struct, sets, time
/usr/lib/python2.6/dist-packages/formless/annotate.py:730: DeprecationWarning: object.__new__() takes no parameters
  rv = cls = InterfaceClass.__new__(cls, name, bases, dct)
/usr/lib/python2.6/dist-packages/nevow/testutil.py:7: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  from popen2 import Popen3
/usr/lib/python2.6/dist-packages/nevow/guard.py:15: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
/usr/lib/pymodules/python2.6/allmydata/storage_client.py:31: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha, time
client node probably started

System is AMD64, dual-core. Tahoe-lafs runs fine.

This tahoe-lafs was installed form apt-get. Let me know if I can supply more informatoin.

[zooko]

Thank you for the bug report!

The deprecation warnings from foolscap have been fixed in foolscap trunk but a new release of foolscap hasn't made containing the fix: ​http://foolscap.lothar.com/trac/ticket/124

The deprecation warnings from nevow (which includes the one from the package named formless in your bug report) I don't know about. I just searched ​the Nevow trac and didn't find evidence that those have been fixed. Could you run tahoe --version and tell me what version of Nevow is in use? I can see from ​the Nevow trac that the latest release of Nevow is v0.10.0.

If these deprecation warnings are still in Nevow v0.10.0 then we should open a ticket with them.

The last one is our fault:

/usr/lib/pymodules/python2.6/allmydata/storage_client.py:31: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha, time

That one is fixed by this patch I just committed: b69e1c600dc7ff7f.

Okay so please see what version of Nevow you have, if it isn't v0.10.0 then upgrade it to v0.10.0 and see if the DeprecationWarnings are still there, and then we'll open a bug report with Nevow.

[imhavoc]

tahoe --version tells me:

allmydata-tahoe: 1.5.0, foolscap: 0.4.2, pycryptopp: 0.5.17, zfec: 1.4.5, Twisted: 8.2.0, Nevow: 0.9.31, zope.interface: unknown, python: 2.6.4, platform: Linux-Ubuntu_9.10-x86_64-64bit_ELF, sqlite: 3.6.16, simplejson: 2.0.9, pyOpenSSL: 0.9, setuptools: 0.6c9, pysqlite: 2.4.1

Is there a graceful way to update all my Python modules?

[zooko]

Opinions differ. I would suggest:

sudo apt-get install python-setuptools
sudo easy_install -U Nevow

Other people might say that this is insanity and they might plead with you to think twice before you throw away all the beauty and perfection of your Ubuntu packaging system.

But don't listen to them.

[imhavoc]

Well, it worked fine. Thank you. apt-get is all fine and dandy until you want to update something outside of their timeline.

[zooko]

So did you sudo easy_install -U Nevow, and when you did, was it v0.10.0 of Nevow that was installed, and if it was, did the Nevow- and formless- related DeprecationWarnings stop happening?

[imhavoc]

Yes, I ran the sudo easy_install -U Nevow command, and it updated Nevow to v0.10.0 flawlessly. That did not stop the DeprecationWarnings, though:

This is what a tahoe restart looks like on the system in question after the update of Nevow:

$ tahoe restart
STOPPING /home/bigpig/.tahoe
process 14341 is dead
STARTING /home/bigpig/.tahoe
/usr/lib/pymodules/python2.6/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated
  import struct, sets, time
/usr/local/lib/python2.6/dist-packages/Nevow-0.10.0-py2.6.egg/formless/annotate.py:730: DeprecationWarning: object.__new__() takes no parameters
  rv = cls = InterfaceClass.__new__(cls, name, bases, dct)
/usr/local/lib/python2.6/dist-packages/Nevow-0.10.0-py2.6.egg/nevow/testutil.py:7: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  from popen2 import Popen3
/usr/local/lib/python2.6/dist-packages/Nevow-0.10.0-py2.6.egg/nevow/guard.py:15: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
/usr/lib/pymodules/python2.6/allmydata/storage_client.py:31: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha, time
client node probably started

[warner]

FYI, the next release of Foolscap will fix the first of those warnings. The change is already in the hg repo, and I'm hoping to cut a release before tahoe-1.6.0 is out.

[zooko]

I updated ​http://divmod.org/trac/ticket/2994#comment:3 explaining that we hope to get this fixed for Lucid. If the Nevow folks (with our help) don't put out a new Nevow release in time and get it into Lucid then I propose that we suppress these warnings in Tahoe-LAFS using the Python warnings module from the stdlib.

[zooko]

I updated ​http://foolscap.lothar.com/trac/ticket/124#comment:6 mentioning that we need a new foolscap release or else suppress this warning in Tahoe-LAFS.

[davidsarah]

Not specific to Ubuntu; I get the same warnings on cygwin.

[zooko]

Foolscap v0.5 is out and it allegedly contains the fix for ​http://foolscap.lothar.com/trac/ticket/124 . If someone could confirm that Tahoe-LAFS emits no DeprecationWarnings about "sets" when run with Foolscap v0.5 then we can consider that issue fixed.

[zooko]

So if Foolscap v0.5.0 gets into Ubuntu 10.04 then that DeprecationWarning will be fixed in 10.04. Here is the issue ticket for Debian requesting that Debian upgrade foolscap: ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=567145 .

For Nevow it doesn't look like it is going to get fixed in Nevow trunk, a new version of Nevow released, and that new version accepted into Ubuntu in time for 10.04, so I think we should suppress the DeprecationWarning in Tahoe-LAFS. There is sample code to do that in the Nevow ticket: ​http://divmod.org/trac/ticket/2994#comment:15

To apply this to Tahoe-LAFS you find the line that emits the warning, which I assume is import nevow, and wrap it in:

import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning, 
    message="object.__new__\(\) takes no parameters", 
    append=True) 
try:
    import nevow
finally:
    warnings.filters.pop()

Also we should update src/allmydata/test/test_runner.py​ to make the code-under-test fail if it emits anything to stderr when the tahoe command is executed. (test_runner.py used to do that, but I made it stop doing that and allow the code-under-test to pass while emitting DeprecationWarnings. Oh, here is the patch: 3a1196701f088a5f.)

[zooko]

This patch suppresses the warnings that result from import nevow and adds back in a unit test that makes sure that the code under test doesn't emit anything to stdout when run in --quiet mode.

[zooko]

Folks: review-needed! This one is very easy to review. The only thing that you should think about is whether importing nevow in src/allmydata/__init__.py​ will cause some weird packaging/loading problem. I can't think of anything. Anyway, as soon as someone says "This is okay" then we'll see how the buildbots like it.

[davidsarah]

This is okay. It does introduce a pyflakes warning, which can be suppressed by

    hush_pyflakes = nevow
    del hush_pyflakes

after import nevow. Also it doesn't suppress the warnings that occur before running unit tests:

c:\python26\lib\site-packages\twisted-8.2.0-py2.6-win32.egg\twisted\persisted\sob.py:12: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import os, md5, sys
c:\python26\lib\site-packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha

[davidsarah]

cli: suppress DeprecationWarnings? emitted from importing nevow and twisted. Fixes #859

[zooko]

applied David-Sarah's version: 6215ebd95e292cca. Thanks!

[davidsarah]

The warnings on starting a node are fixed, but after the 1.6 release we should also fix the ones that occur on running setup.py trial (see comment:16 above). I don't know where to put the warning suppression code to do that.

[zooko]

I think these are all fixed. Let's not worry about suppressing the warnings from Twisted-8.2.0 since newer versions of Twisted have already been released that don't emit those warnings.