[tahoe-dev] building and pre-existing dependencies

[Greg Troxel]

  > I am trying to package tahoe for pkgsrc, the native packaging system on
  > NetBSD and Dragonfly, also portable to Darwin, Linux, Solaris, OpenBSD,
  > FreeBSD, IRIX, AIX, and a few others.

  Sweet! I would be happy to help.

Thanks very much - your egg hint is I think what I needed.

From a packager viewpoint, it would be nice if tahoe's setup.py had an
option --no-recurse (name not important) to tell it to fail instead of
trying to download and build other packages.  I can see why the
automatic behavior is useful for users but it works against bringing the
whole build process under CM (sources identified, repeatable builds, no
net use during build, etc.)  I'd rather have it fail and then go package
the other python dependency and depend on it.

  > I don't understand how setup.py decides if a particular python package
  > is already present or needs to be downloaded and built.

  In theory, it looks for a file or directory named
  $DISTRIBUTIONNAME-$VERSION-py$PYTHONVERSION.egg-info in any directory
  which is on the Python sys.path. In practice there are a few known
  bugs in different versions of different tools which might cause it to
  do this re-download behavior that you observed even if the .egg-info
  file is present.

Yes, that is exactly the issue.

  An import thing to keep in mind is that "Crypto" is the name of a
  Python module (which is sometimes confusingly called a Python
  "package") i.e., something that can be imported. "pycrypto" is the
  name of the Python package (which is always confusingly called a
  Python "distribution") i.e. something that can be installed onto a
  system and that another distribution (package) can depend on.

  So, to test whether the Python "distribution" is installed under the
  expected name and version number it doesn't help to import Crypto,
  which only tells you whether the Python module is importable. To see
  whether the Python "distribution" is installed you have to do this:

Wow - that makes sense but is not obvious from anything I'd managed to
read so far.

  python -c 'import pkg_resources;print pkg_resources.require("pycrypto")'

That failed, with my security/py-crypto package.  py-asn1 uses egg.mk
and seems ok
(http://www.netbsdpkgsrc.info/packages/security/py-asn1.html).  I had
used extension.mk instead of egg.mk because of trouble with
--single-version-externally-managed.

  Now you could if you want patch the Tahoe-LAFS setup.py to not use
  this mechanism at all and just rely on the pkgsrc dependency
  mechanism. That would be okay, but it would be one more patch you'd
  have to maintain, and other Python tools besides the Tahoe-LAFS
  setup.py might likewise be depending on the .egg-info. So it might be
  good to get to the bottom of why there isn't a
  pycrypto-$VER-py$PYVER.egg-info on your sys.path.

Agreed; the missing egg-info is the real bug, and other such files have
been placed there by pkgsrc.  Apparently there is something odd about
py-crypto.

gdt 670 /usr/pkg/lib/python2.6/site-packages > l -d *egg*
drwxr-xr-x  2 root  wheel    512 Jul 20 19:59 Twisted-10.0.0-py2.6.egg-info
drwxr-xr-x  2 root  wheel    512 Jul 16 15:43 bsddb3-5.0.0-py2.6.egg-info
drwxr-xr-x  2 root  wheel    512 Jul 20 19:59 pyasn1-0.0.11a-py2.6.egg-info
-rw-r--r--  1 root  wheel    623 Jul 23 13:45 pycrypto-2.1.0-py2.6.egg-info
drwxr-xr-x  2 root  wheel    512 Jul 22 16:24 pycryptopp-0.5.19-py2.6.egg-info
drwxr-xr-x  2 root  wheel    512 Jul 16 15:43 setuptools-0.6c11-py2.6.egg-info
drwxr-xr-x  2 root  wheel    512 Jul 20 19:58 zope.interface-3.6.1-py2.6.egg-info

I now think there are two kinds of egg-aware packages: 1st-class ones
that take --single-version-externally-managed, and older ones that just
install an egg file, and that pkgsrc's egg.mk is only about the first
kind.

Here's my question to the pkgsrc technical list:


  From: Greg Troxel <gdt at ir.bbn.com>
  Subject: python egg info
  To: tech-pkg at netbsd.org
  Date: Fri, 23 Jul 2010 14:07:24 -0400

  I am packaging some python modules/packages in order to have
  dependencies for tahoe-lafs, and finding egg metadata file usage in
  pkgsrc confusing.  The particular package in question is
  security/py-crypto.

  After including lang/python/egg.mk, 'make package' (DESTDIR mode) fails
  with:

    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help

    error: option --single-version-externally-managed not recognized

  If I comment out that argument in egg.mk, the package builds, but then I have
  no installed egg info.

  In python26/patches/patch-av, installation of egg metadata files is
  disabled unless PY_NO_EGG is defined to other than yes.  Setting it to
  NO in the package causes egg metadata to be installed.  It seems the
  python way is to install these files so that the python requirement
  system works.  There are other egg metadata files installed.

  With removing --single-version-externally-managed and setting PY_NO_EGG
  to NO, I get an egg metadata file installed and require works:

  > python2.6 -c 'import pkg_resources;print pkg_resources.require("pycrypto")'
  [pycrypto 2.1.0 (/usr/pkg/lib/python2.6/site-packages)]


  So,

  1) Why does PY_NO_EGG default to yes?  Should it be set to NO in egg.mk?
  Or is that about not installing egg metadata in the package directory
  itself?

  2) Is a python module that has setup.py and doesn't implement
  --single-version-externally-managed broken?

  My tentative conclusions are:

  A) that --single-version-externally-managed is the python way for an
  egg-aware package to be told

    1) there will only be one version of this package installed, and

    2) the egg data should be in .../site-package so that it will be found
    by require

  B) py-crypto is not really 'egg aware' in the modern mainstream sense,
  but in some more limited sense that still allows require to find it.

  Do we need to all EGG_OLD_STYLE=YES to cause egg.mk to

    not add --single-version-externally-managed

    not set PY_NO_EGG to YES

  Or perhaps there's another approach - I think the odds my understanding
  is correct is only about 50%.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20100723/cd8a2cb6/attachment.pgp>