Debian package: Missing dependency on Debian/etch

[[4-tea-2]]

Starting tahoe after upgrading a Debian/etch system to use the Debian Tahoe package 1.4.1-r3916 fails with this error message:

tahoe@{hostname}:~$ tahoe start
Traceback (most recent call last):
  File "/usr/bin/tahoe", line 2, in <module>
    from allmydata.scripts import runner
  File "/usr/lib/python2.5/site-packages/allmydata/scripts/runner.py", line 9, in <module>
    pkg_resources.require('allmydata-tahoe')
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 524, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: pysqlite>=2.0.5

Perhaps including python-pysqlite2 as a dependency in debian/control would fix this?

[[4-tea-2]]

Sorry, I forgot to mention that installing python-pysqlite2 manually fixed the problem.

[warner]

Hrm, on python2.5, it's not supposed to declare a dependency on pysqlite (since 2.5 comes with sqlite in the standard library).

Zooko?

[warner]

#728 was for adding this change. I grepped the current source code and the pysqlite requirement is guarded by an if sys.version_info < (2,5). Could you make sure that sys.version_info looks correct on your system, and that this conditional is doing the right thing?

[zooko]

Assigning to [4-tea-2] to investigate.

[warner]

Incidentally, when Zooko asked me about adding this dependency declaration, I agreed to it only with the condition that it not cause exactly this sort of problem (setuptools has taught me to not trust its dependency calculations: I am constantly seeing it rebuild things which are already installed). If this turns out to be some sort of deep problem, I'll recommend that we remove the declaration.

[zooko]

setuptools installs requirements that are already installed (#657 ("python ./setup.py test" rebuilds packages), #717 (unnecessary rebuild of dependencies when tahoe-deps/ is present)), but I don't think I've seen it install something which wasn't required. My guess is that [4-tea-2] installed a .deb which didn't declare a dependency on sqlite2 (possibly because the .deb is in error? Or because it was a .deb for python 2.6?), and then ran it with Python 2.5.

[4-tea-2]: which .deb of Tahoe-LAFS did you install?

Thanks!

[zooko]

Formatted original report with wiki quoting.

[zooko]

Oh, when I wrote comment:7, I was thinking that Python >= 2.6 comes with sqlite, but it is actually Python >= 2.5. [4-tea-2], would you please run the following and post the results?

PYTHONPATH=./support/lib/python2.5/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")'

? For example, here are the results on my machine:

$ PYTHONPATH=./support/lib/python2.4/site-packages/:$PYTHONPATH python -c 'import pkg_resources;print pkg_resources.require("allmydata-tahoe")'
[allmydata-tahoe 1.4.1-r3914 (/home/zooko/playground/allmydata/tahoe/trunk/src), pysqlite 2.5.5 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/pysqlite-2.5.5-py2.4-solaris-2.11-i86pc.egg), Nevow 0.9.32 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/Nevow-0.9.32-py2.4.egg), foolscap 0.4.1 (/usr/lib/python2.4/site-packages/foolscap-0.4.1-py2.4.egg), Twisted 8.2.0 (/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-solaris-2.11-i86pc.egg), zope.interface 3.5.1 (/usr/lib/python2.4/site-packages/zope.interface-3.5.1-py2.4-solaris-2.11-i86pc.egg), simplejson 2.0.7 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/simplejson-2.0.7-py2.4-solaris-2.11-i86pc.egg), zfec 1.4.4 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/zfec-1.4.4-py2.4-solaris-2.11-i86pc.egg), pycryptopp 0.5.4 (/usr/lib/python2.4/site-packages/pycryptopp-0.5.4-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), pyOpenSSL 0.8 (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/pyOpenSSL-0.8-py2.4-solaris-2.11-i86pc.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), pyutil 1.3.26 (/usr/lib/python2.4/site-packages/pyutil-1.3.26-py2.4.egg), argparse 0.8.0 (/usr/lib/python2.4/site-packages/argparse-0.8.0-py2.4.egg), setuptools 0.6c12dev (/home/zooko/playground/allmydata/tahoe/trunk/support/lib/python2.4/site-packages/setuptools-0.6c12dev.egg), zbase32 1.1.0 (/usr/lib/python2.4/site-packages/zbase32-1.1.0-py2.4.egg)]

Then, please look at the _auto_deps.py file which is in the allmydata-tahoe package. For example, on my machine it is:

$ grep -A1 -B1 -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata/_auto_deps.py

# Sqlite comes built into Python >= 2.5, and is provided by the "pysqlite"
# distribution for Python 2.4.
--
if sys.version_info < (2, 5):
    # pysqlite v2.0.5 was shipped in Ubuntu 6.06 LTS "dapper" and Nexenta NCP 1.
    install_requires.append("pysqlite >= 2.0.5")

The _auto_deps.py ought to be the same as the current trunk version: _auto_deps.py.

Finally, please look at the requires.txt file in the .egg-info, like this:

$ grep -i sqlite /home/zooko/playground/allmydata/tahoe/trunk/src/allmydata_tahoe.egg-info/requires.txt 
pysqlite >= 2.0.5

This shows that on my machine pysqlite appears in the requires.txt, because the .egg-info was built using Python 2.4. If you built with Python 2.5 then pysqlite should not appear in your .egg-info. You could try rm'rf the whole allmydata_tahoe.egg-info directory and rebuilding.

Thanks!

[zooko]

Ooh, I get it. [4-tea-2] didn't build the .egg-info himself, it came with the .deb that we built. So, we have to decide whether we want our .deb to depend on python-pysqlite, or whether we want it to not completely support Python 2.4 (or we could even build different .deb's for each Python version). I'm okay with either of those two options. (Not so much with the Python-version-specific debs option.)

I guess that the build machine where the Debian packages are being created is using Python 2.4, so pysqlite is being added to the .egg-info's requires.txt.

Assigning to Brian to tell me what our policy is about .debs and pysqlite.

[warner]

Sigh. So, building separate .debs for py2.4/py2.5/py2.6 is painful (nobody does it anymore, now that debian-python-policy generates .pycs at install time). And I don't believe the .deb control-file "Depends:" format isn't rich enough to express (py>=2.5 or pysqlite).

On the other hand, the "standard" version of python is well-known per-platform (it's 2.4 in the older ones, 2.5 in the newer ones, and I guess Ubuntu Karmic will probably have 2.6). So we could say that when we build a .deb on a platform with /usr/bin/python=2.4, we declare the dependency upon pysqlite, and when we build it on a py2.5-using platform, we don't.

To implement this, we should update the misc/*/debian/control files to declare the pysqlite dependency on the older platforms, and leave the newer ones alone (not declaring it).

This also means that we should probably stop using the feisty rules for all platforms.. we at least need to identify the release for which python switched to 2.5 and use a newer set of rules for the 2.5-using set.

I'll try to look at this today, but I'm not sure I'll get the time.

[warner]

Ok, I think I understand this better now.

• debian/etch ships with python2.4 as the standard /usr/bin/python (​http://packages.debian.org/etch/python).
• our most recent tahoe .deb for etch (​http://allmydata.org/debian/dists/etch/tahoe/binary-i386/allmydata-tahoe_1.4.1-r3916_all.deb) was built on an Etch system with python2.4
  • therefore its _auto_deps.py correctly declares a requirement on pysqlite, enforced at runtime, via the call to require_auto_deps() in src/allmydata/__init__.py
  • the .deb's .egg-info/requires.txt correctly includes a line which requires pysqlite >= 2.0.5, which will probably be enforced by anything that goes poking around .egg-info . In particular, the code that was added to src/allmydata/scripts/runner.py (the main CLI entry point) to call pkg_requires.require('allmydata-tahoe') will read the .egg-info data and enforce its requirements.
• however, the packaging (i.e. debian/control) does not declare this requirement. This is a bug. Installing just this .deb (and no extra non-Depends-required packages) and then running the supplied /usr/bin/tahoe with python2.4 should fail with the same sort of exception displayed, because _auto_deps.py insists on having pysqlite, but APT did not know that python-pysqlite2 should have been installed.
• But, 4-tea-2 appears to be using python2.5 to start the Tahoe node on this etch system, because the traceback listed above shows /usr/lib/python2.5/site-packages. Etch includes a python2.5 package (​http://packages.debian.org/etch/python2.5), but it does not touch /usr/bin/python, so either 4-tea-2 has set up a symlink or installed python2.5 outside the awareness of APT (perhaps directly to /usr/local/bin). This is what was confusing me.
  • the .deb provides an allmydata_tahoe-1.4.1_r3916-py2.4.egg-info/ directory, and does not provide any .egg-info data for python2.5 . I don't know how setuptools reacts to this: would it refuse to believe that tahoe was actually installed when run under py2.5? If so, then I'm still confused as to where 4-tea-2's tahoe was finding the .egg-info data.
  • when run under python2.5, the code in _auto_deps.py should *not* require pysqlite
  • however the .egg-info line (generated on our Etch buildslave) still requires it, and thus the scripts/runner.py call will demand it.

Why is that pkg_resources.require('allmydata-tahoe') line there? If it weren't, then the correct runtime-computed dependencies in _auto_deps.py would be enforced, and the incorrect package-build-time dependencies in .egg-info/requires.txt would not.

Now, it happens that the .deb packaging for this is incorrect, and the Etch .deb should declare a dependency on python-sqlite2 because that's needed by the default version of python on an Etch system, and it happens that fixing the packaging would also happen to make this problem go away. But it's still wrong: this is a dependency that changes depending upon the version of Python used to run bin/tahoe, and recording the value that was computed at package-build time seems inappropriate. It should be possible to run tahoe under py2.5 on this system without having pysqlite installed.

[zooko]

I guess a quicker solution is just to have the allmydata-tahoe .deb always depend on python-sqlite. While we're at it, I can swap the order of attempted-import in src/allmydata/__init__.py​ and src/allmydata/scripts/backupdb.py​, so that whatever python-sqlite package that is installed is the one that actually gets used (in preference to the sqlite module that comes in the Python standard library).

[zooko]

My comment:13 was in reply to your comment:11, not your comment:12.

[warner]

I updated the debian packaging rules last night, and now the py2.4-based distributions (etch and edgy, I believe) add a python-sqlite2 dependency, and the more recent ones do not. I'd prefer to not force py2.5-based platforms to install an extra package. And I'd prefer that we use the stdlib sqlite3 module in preference to an external pysqlite2 module. Think about it this way: when py2.4 is long dead, do you still want us to be requiring people to install a third-party module to duplicate stdlib functionality?

[zooko]

Thanks, Brian!