Missing `allmydata/web/static/img` subdirectory when installed from `pip`.

[nejucomo]

Synopsis:

The front page ​welcome.xhtml has ​an image link for /img/logo.png which does not exist in a pip installed tahoe. In fact, the entire allmydata/web/static/img subdirectory is absent:

$ cd $VIRTUAL_ENV
$ find . -name '*.png'
./lib/python2.7/site-packages/allmydata/web/static/icon.png
$ ls ./lib/python2.7/site-packages/allmydata/web/static/
css  d3-2.4.6.min.js  d3-2.4.6.time.min.js  download_status_timeline.js  icon.png  jquery-1.6.1.min.js  tahoe.css
$ ./bin/tahoe --version
allmydata-tahoe: 1.10.0
foolscap: 0.6.4
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
zfec: 1.4.24
Twisted: 13.2.0
Nevow: 0.10.0
zope.interface: unknown
python: 2.7.5+
platform: Linux-debian_jessie/sid-x86_64-64bit_ELF
pyOpenSSL: 0.14
simplejson: 3.3.3
pycrypto: 2.6.1
pyasn1: 0.1.7
mock: 1.0.1
setuptools: 0.9.8

Warning: dependency 'cryptography' (version '0.3') found by pkg_resources not found by import.
Warning: dependency 'six' (version '1.6.1') found by pkg_resources not found by import.
Warning: dependency 'cffi' (version '0.8.2') found by pkg_resources not found by import.
Warning: dependency 'pycparser' (version '2.10') found by pkg_resources not found by import.

For debugging purposes, the PYTHONPATH was
  '/home/n/lib/python:/usr/lib/pyshared/python2.7/'
install_requires was
  ['setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface == 3.6.0, == 3.6.1, == 3.6.2, >= 3.6.5', 'Twisted >= 11.0.0', 'foolscap >= 0.6.3', 'pyOpenSSL', 'Nevow >= 0.6.]
sys.path after importing pkg_resources was
  /home/n/virtualenvs/lafs-wapi-dev/bin:
  /home/n/lib/python:
  /usr/lib/pyshared/python2.7:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7/plat-x86_64-linux-gnu:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7/lib-tk:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7/lib-old:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7/lib-dynload:
  /usr/lib/python2.7:
  /usr/lib/python2.7/plat-x86_64-linux-gnu:
  /usr/lib/python2.7/lib-tk:
  /home/n/virtualenvs/lafs-wapi-dev/local/lib/python2.7/site-packages

Details:

The image link was added in ​changeset 709be93a29e20026e61a436dac3fa1c160e9cef2, which also contains the image.

I checked to see if this was a case of build misconfiguration, and it appears that the image should be included in the ​MANIFEST.in.

I also verified that an sdist tarball generated from master has the file:

$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
$ git rev-parse HEAD
0ef593947755a8b81edc73d033219724268faf80
$ rm -rf ./dist/
$ python ./setup.py sdist | grep logo
warning: no previously-included files matching '*~' found anywhere in distribution
hard linking src/allmydata/web/static/img/logo.png -> allmydata-tahoe-1.10.0.post76/src/allmydata/web/static/img
$ tar -tzf ./dist/allmydata-tahoe-1.10.0.post76.tar.gz | grep logo.png
allmydata-tahoe-1.10.0.post76/src/allmydata/web/static/img/logo.png

At this point I am uncertain how the pip package allmydata-tahoe is generated. It seems like a potential next step is to create a packaging test within the buildbot configuration to verify that this file is included in the allmydata-tahoe package distribution.

[daira]

I see the problem. allmydata.web.static.img should be included in the list of packages here: setup.py#L447​. I will fix it.

[daira]

Note that MANIFEST.in is a generated file. The .img files are supposed to be included as a result of this line:

recursive-include src *.xhtml *.js *.png *.css

[daira]

The sdist installed by pip is created using python setup.py sdist --formats=gztar (or similar).

[daira]

​https://github.com/tahoe-lafs/tahoe-lafs/pull/87

[daira]

(2014-03-29 00:14:03) daira1: dstufft: the only difference between the sdist that is installed correctly by pip (including the src/allmydata/web/static/img directory) and the sdist that is not installed correctly
(2014-03-29 00:15:07) daira1: is that the former has 'allmydata.web.static.img', in the 'packages=' argument to setuptools.setup(...)
(2014-03-29 00:15:52) daira1: however the 'package_data=' argument contains "allmydata.web.static.img": ["*.png"] in both cases
(2014-03-29 00:16:01) daira1: is this a bug in pip and/or setuptools?
(2014-03-29 00:16:30) dstufft: can you link me to the setup.py?
(2014-03-29 00:16:39) dstufft: or the two setup.py's
(2014-03-29 00:16:40) daira1: yes, just a sec
(2014-03-29 00:19:50) daira1: with 'allmydata.web.static.img': https://github.com/tahoe-lafs/tahoe-lafs/blob/1931505518726256ff2e8becb5634910afbcf766/setup.py
(2014-03-29 00:20:09) daira1: without: https://github.com/tahoe-lafs/tahoe-lafs/blob/master/setup.py
(2014-03-29 00:20:47) daira1: diff: https://github.com/tahoe-lafs/tahoe-lafs/commit/1931505518726256ff2e8becb5634910afbcf766
(2014-03-29 00:21:56) daira1: and here's the ticket: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2209
(2014-03-29 00:22:45) daira1: I was under the impression that packages= was only supposed to contain source packages? I could be wrong about that
(2014-03-29 00:25:33) dstufft: daira1: allmydata.web.static.img is a package, it has a __init__.py
(2014-03-29 00:26:15) daira1: it has an __init__.py only to work around a setuptools bug
(2014-03-29 00:26:38) dstufft: in order for that to work without allmydata.web.static.img in packages=[...] you'd need to set package_data={"allmydaa.web.static": ["img/*.png"]}
(2014-03-29 00:26:59) daira1: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1960
(2014-03-29 00:27:02) dstufft: in other words, the keys for package_data has to be installed packages
(2014-03-29 00:27:19) daira1: ah, I see
(2014-03-29 00:28:06) daira1: currently we have:
(2014-03-29 00:28:06) daira1:       package_data={"allmydata.web": ["*.xhtml"],
(2014-03-29 00:28:06) daira1:                     "allmydata.web.static": ["*.js", "*.png", "*.css"],
(2014-03-29 00:28:06) daira1:                     "allmydata.web.static.css": ["*.css"],
(2014-03-29 00:28:06) daira1:                     "allmydata.web.static.img": ["*.png"],
(2014-03-29 00:28:06) daira1:                     },
(2014-03-29 00:28:17) daira1: so that should be:
(2014-03-29 00:28:40) daira1:       package_data={"allmydata.web": ["*.xhtml"],
(2014-03-29 00:28:40) daira1:                     "allmydata.web.static": ["*.js", "*.png", "*.css", "img/*.png", "css/*.css"],
(2014-03-29 00:28:40) daira1:                     },
(2014-03-29 00:28:41) daira1: ?
(2014-03-29 00:28:47) dstufft: Yea I think so
(2014-03-29 00:29:02) dstufft: I'm not 100% sure on that, but I think so
(2014-03-29 00:29:08) daira1: I'll try it
(2014-03-29 00:29:10) daira1: thanks
(2014-03-29 00:29:13) dstufft: np
(2014-03-29 00:29:16) dstufft: let me know!
(2014-03-29 00:29:22) ***daira1 nods
(2014-03-29 00:30:57) daira1: hmm, or maybe:
(2014-03-29 00:30:57) daira1:       package_data={"allmydata.web": ["*.xhtml", "static/*.js", "static/*.png",
(2014-03-29 00:30:57) daira1:                                       "static/*.css", "static/img/*.png", "static/css/*.css"],
(2014-03-29 00:30:57) daira1:                     },

The pull request uses the latter. I smoke-tested it with pip and also using:

make clean
python setup.py build
sudo python setup.py install

[daira]

I also checked that this doesn't regress #1960. (zooko was exactly correct in 1960#comment:10.)

[zooko]

Here's a relevant patch: b09b400c8a38ac7b1e1a8d788376db8cce666660/trunk, pull request: ​https://github.com/tahoe-lafs/tahoe-lafs/commit/b09b400c8a38ac7b1e1a8d788376db8cce666660 and buildbot run: ​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/263

So can we close this ticket now, Daira?

[zooko]

Replying to daira:

Note that MANIFEST.in is a generated file.

No, MANIFEST is generated. MANIFEST.in​ is a source file.

[daira]

Replying to zooko:

Replying to daira:

Note that MANIFEST.in is a generated file.

No, MANIFEST is generated. MANIFEST.in​ is a source file.

You're right, sorry.