apparent serious integrity problem in build system - setuptools bug?

Sun Mar 16 00:47:13 UTC 2014

It's quarterly branch time at pkgsrc, so I'm paying attention to bulk
build reports, and notice that tahoe-lafs 1.10.0 failed to build on
SmartOS (which is more or less OpenSolaris).  The log looked odd, and I
then noticed that it failed on NetBSD 6, which is the primary platform
I've used to debug the pkgsrc package.

So, looking further, I see that the build is fetching data from
tahoe.org, and using that to install additional modules.  If that's
really what's happening, it's completely ridiculous from a  security
viewpoint - the distribution tarball that's been audited (or not, I
know) with a particular hash should build the same binary.

More troubling, I can't believe that I wouldn't have noticed this before.

But, I know setuptools has been updated recently in pkgsrc, from 1.3 to
3.1!

Has anyone else seen this?

I've attached my build log

----------------------------------------
=> Bootstrap dependency digest>=20010302: found digest-20121220
=> Checksum SHA1 OK for allmydata-tahoe-1.10.0.tar.bz2
=> Checksum RMD160 OK for allmydata-tahoe-1.10.0.tar.bz2
===> Installing dependencies for tahoe-lafs-1.10.0nb2
==========================================================================
The following variables will affect the build process of this package,
tahoe-lafs-1.10.0nb2.  Their current value is shown below:

        * PYTHON_VERSION_DEFAULT = 27

Based on these variables, the following variables have been set:

        * PYPACKAGE = python27

You may want to abort the process now with CTRL-C and change their value
before continuing.  Be sure to run `/usr/bin/make clean' after
the changes.
==========================================================================
=> Tool dependency gmake>=3.81: found gmake-4.0
=> Tool dependency ccache-[0-9]*: found ccache-3.1.9
=> Tool dependency checkperms>=1.1: found checkperms-1.11
=> Build dependency py27-darcsver-[0-9]*: found py27-darcsver-1.7.4
=> Full dependency py27-zfec-[0-9]*: found py27-zfec-1.4.22
=> Full dependency py27-zbase32-[0-9]*: found py27-zbase32-1.1.5
=> Full dependency py27-simplejson-[0-9]*: found py27-simplejson-3.3.2
=> Full dependency py27-sqlite3-[0-9]*: found py27-sqlite3-2.7.6nb2
=> Full dependency py27-argparse-[0-9]*: found py27-argparse-1.2.1
=> Full dependency py27-pyutil-[0-9]*: found py27-pyutil-1.9.3
=> Full dependency py27-mock-[0-9]*: found py27-mock-1.0.1
=> Full dependency py27-foolscap>=0.6.3: found py27-foolscap-0.6.4
=> Full dependency py27-twisted-[0-9]*: found py27-twisted-13.2.0
=> Full dependency py27-asn1-[0-9]*: found py27-asn1-0.1.7
=> Full dependency py27-crypto-[0-9]*: found py27-crypto-2.6.1nb1
=> Full dependency py27-cryptopp-[0-9]*: found py27-cryptopp-0.6.0
=> Full dependency py27-OpenSSL-[0-9]*: found py27-OpenSSL-0.14
=> Full dependency py27-nevow-[0-9]*: found py27-nevow-0.10.0nb2
=> Full dependency py27-setuptools>=0.6c9: found py27-setuptools-3.1
=> Full dependency python27>=2.7.1nb2: found python27-2.7.6nb3
===> Overriding tools for tahoe-lafs-1.10.0nb2
===> Extracting for tahoe-lafs-1.10.0nb2
===> Patching for tahoe-lafs-1.10.0nb2
===> Creating toolchain wrappers for tahoe-lafs-1.10.0nb2
===> Configuring for tahoe-lafs-1.10.0nb2
=> Replacing python interpreter in src/allmydata/reliability.py src/allmydata/storage/shares.py.
WARNING: [replace-interpreter] Skipping non-existent file "src/allmydata/reliability.py".
=> Checking for portability problems in extracted files
===> Building for tahoe-lafs-1.10.0nb2
(cd /u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/allmydata-tahoe-1.10.0/ && /usr/bin/env USETOOLS=no PTHREAD_CFLAGS=\ -pthread\  PTHREAD_LDFLAGS=\ -pthread PTHREAD_LIBS= PTHREADBASE=/usr DL_CFLAGS=\ -pthread\  DL_LDFLAGS=\ -pthread DL_LIBS= PYTHON=/usr/pkg/bin/python2.7 CC=gcc CFLAGS=-O2\ -I/usr/include\ -I/usr/pkg/include CPPFLAGS=-I/usr/include\ -I/usr/pkg/include CXX=c++ CXXFLAGS=-O2\ -I/usr/include\ -I/usr/pkg/include COMPILER_RPATH_FLAG=-Wl,-R F77=g77 FC=g77 FFLAGS=-O LANG=C LC_ALL=C LC_COLLATE=C LC_CTYPE=C LC_MESSAGES=C LC_MONETARY=C LC_NUMERIC=C LC_TIME=C LDFLAGS=-L/usr/lib\ -Wl,-R/usr/lib\ -L/usr/pkg/lib\ -Wl,-R/usr/pkg/lib LINKER_RPATH_FLAG=-R PATH=/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.wrapper/bin:/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.buildlink/bin:/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.ccache/bin:/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.gcc/bin:/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.tools/bin:/usr/pkg/bin:/home/gdt/bin:/home/gdt/bin/i386-NetBSD:/usr/y0/sbin:/usr/y0/bin:/usr/pkg/sbin:/usr/pkg/bin:/usr/X11R7/bin:/usr/local/sbin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/pkg/bin:/usr/pkg/bin PREFIX=/usr/pkg MAKELEVEL=0 PKG_SYSCONFDIR=/usr/pkg/etc CXXCPP=cpp HOME=/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.home CPP=cpp LINK_ALL_LIBGCC_HACK= LOCALBASE=/usr/pkg NO_WHOLE_ARCHIVE_FLAG=-Wl,--no-whole-archive WHOLE_ARCHIVE_FLAG=-Wl,--whole-archive X11BASE=/usr/pkg X11PREFIX=/usr/pkg PKGMANDIR=man PKGINFODIR=info PKGGNUDIR=gnu/ MAKECONF=/dev/null OBJECT_FMT=ELF USETOOLS=no BSD_INSTALL_PROGRAM=/usr/bin/install\ -c\ -s\ -o\ gdt\ -g\ ir\ -m\ 755 BSD_INSTALL_SCRIPT=/usr/bin/install\ -c\ -o\ gdt\ -g\ ir\ -m\ 755 BSD_INSTALL_LIB=/usr/bin/install\ -c\ -o\ gdt\ -g\ ir\ -m\ 755 BSD_INSTALL_DATA=/usr/bin/install\ -c\ -o\ gdt\ -g\ ir\ -m\ 644 BSD_INSTALL_MAN=/usr/bin/install\ -c\ -o\ gdt\ -g\ ir\ -m\ 644 BSD_INSTALL=/usr/bin/install BSD_INSTALL_PROGRAM_DIR=/usr/bin/install\ -d\ -o\ gdt\ -g\ ir\ -m\ 755 BSD_INSTALL_SCRIPT_DIR=/usr/bin/install\ -d\ -o\ gdt\ -g\ ir\ -m\ 755 BSD_INSTALL_LIB_DIR=/usr/bin/install\ -d\ -o\ gdt\ -g\ ir\ -m\ 755 BSD_INSTALL_DATA_DIR=/usr/bin/install\ -d\ -o\ gdt\ -g\ ir\ -m\ 755 BSD_INSTALL_MAN_DIR=/usr/bin/install\ -d\ -o\ gdt\ -g\ ir\ -m\ 755 BSD_INSTALL_GAME=/usr/bin/install\ -c\ -s\ -o\ gdt\ -g\ ir\ -m\ 2555 BSD_INSTALL_GAME_DATA=/usr/bin/install\ -c\ -o\ gdt\ -g\ ir\ -m\ 664 BSD_INSTALL_GAME_DIR=/usr/bin/install\ -d\ -o\ gdt\ -g\ ir\ -m\ 775 INSTALL_INFO= MAKEINFO=/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.tools/bin/makeinfo FLEX= BISON= PKG_CONFIG= PKG_CONFIG_LIBDIR=/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.buildlink/lib/pkgconfig:/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.buildlink/share/pkgconfig PKG_CONFIG_LOG=/u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/.pkg-config.log PKG_CONFIG_PATH= WRAPPER_DEBUG=no WRAPPER_UPDATE_CACHE=yes VIEWBASE=/usr/pkg /usr/pkg/bin/python2.7  setup.py  build )
running update_version
no version-control data found, leaving _version.py alone
running develop
Not found: tahoe-deps
Not found: ../tahoe-deps
running egg_info
writing requirements to src/allmydata_tahoe.egg-info/requires.txt
writing src/allmydata_tahoe.egg-info/PKG-INFO
writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt
writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt
writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt
package init file 'src/allmydata/web/static/__init__.py' not found (or not a regular file)
package init file 'src/allmydata/web/static/css/__init__.py' not found (or not a regular file)
reading manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*~' found anywhere in distribution
writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
running build_ext
Creating /u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/site.py
Processing setuptools-0.6c16dev4.egg
Copying setuptools-0.6c16dev4.egg to /u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages
Adding setuptools 0.6c16dev4 to easy-install.pth file
Installing easy_install_z-2.6 script to support/bin
Installing easy_install_z script to support/bin

Installed /u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/setuptools-0.6c16dev4.egg
Creating /u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/allmydata-tahoe.egg-link (link to src)
Adding allmydata-tahoe 1.10.0 to easy-install.pth file
Installing tahoe script to support/bin

Installed /u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/allmydata-tahoe-1.10.0/src
Processing dependencies for allmydata-tahoe==1.10.0
Searching for six>=1.5.2
Reading https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-sdists/
Reading https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Reading http://pypi.python.org/simple/six/
Best match: six 1.6.1
Downloading https://pypi.python.org/packages/source/s/six/six-1.6.1.tar.gz#md5=07d606ac08595d795bf926cc9985674f
Processing six-1.6.1.tar.gz
Running six-1.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-oD0SEu/six-1.6.1/egg-dist-tmp-OaLoJS
no previously-included directories found matching 'documentation/_build'
zip_safe flag not set; analyzing archive contents...
six: module references __file__
six: module references __path__
Adding six 1.6.1 to easy-install.pth file

Installed /u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs/work/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/six-1.6.1-py2.7.egg
Searching for cryptography>=0.2.1
Reading http://pypi.python.org/simple/cryptography/
Best match: cryptography 0.2.2
Downloading https://pypi.python.org/packages/source/c/cryptography/cryptography-0.2.2.tar.gz#md5=f002a442c8c5c7463bf8d2f11f6c3128
Processing cryptography-0.2.2.tar.gz
Running cryptography-0.2.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-YFuWag/cryptography-0.2.2/egg-dist-tmp-ii0OcJ
Searching for six>=1.4.1
Reading http://pypi.python.org/simple/six/
Best match: six 1.6.1
Downloading https://pypi.python.org/packages/source/s/six/six-1.6.1.tar.gz#md5=07d606ac08595d795bf926cc9985674f
Processing six-1.6.1.tar.gz
Running six-1.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-YFuWag/cryptography-0.2.2/temp/easy_install-usM2h9/six-1.6.1/egg-dist-tmp-niM1ew
no previously-included directories found matching 'documentation/_build'
zip_safe flag not set; analyzing archive contents...
six: module references __file__
six: module references __path__

Installed /tmp/easy_install-YFuWag/cryptography-0.2.2/six-1.6.1-py2.7.egg
Searching for cffi>=0.8
Reading http://pypi.python.org/simple/cffi/
Best match: cffi 0.8.2
Downloading https://pypi.python.org/packages/source/c/cffi/cffi-0.8.2.tar.gz#md5=37fc88c62f40d04e8a18192433f951ec
Processing cffi-0.8.2.tar.gz
Running cffi-0.8.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-YFuWag/cryptography-0.2.2/temp/easy_install-UIy6Wk/cffi-0.8.2/egg-dist-tmp-fZ3PnC
error: Permission denied
*** Error code 1

Stop.
make: stopped in /u0/n0/gdt/NetBSD-current/pkgsrc/filesystems/tahoe-lafs
*** Error code 1

Stop.
make: stopped in /usr/pkgsrc/filesystems/tahoe-lafs

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 180 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20140315/8de70adb/attachment.pgp>