setuptools/easy_install/pip-installed package lacks documentation

[nejucomo]

I am a happy user of the pip installability feature, whereby I run "pip install allmydata-tahoe" to get a running tahoe installation.

This installation lacks documentation, at least where I sometimes find other python documentation in $VIRTUALENV_ROOT/share/doc/$PACKAGE_NAME.

Request: Fix the pypi distribution to bundle all documentation such that pip-installed tahoe provides documentation in this location.

If documentation is installed elsewhere, ensure that it is compatible with virtualenv. Specifically I should be able to install different versions of tahoe-lafs in different virtualenvs and have the correct copy of documentation for each install.

[nejucomo]

Note that the title of this ticket is misleading. This has nothing to do with pip. I added the pypi keyword, but this also is not specific to Tahoe-LAFS's ​PyPI page.

I just verified that if I download the ​pypi tarball or the ​self-hosted zip file, and then run:

$ python ./setup.py build
$ python ./setup.py install

-there are no doc files installed outside of the tarball/zipfile contents.

[nejucomo]

I just added the proposed enhancement #2031 which is to serve ./docs from the web gateway. That would necessarily solve this issue by bundling all of ./docs as a package resource directory.

[nejucomo]

I believe the summary should actually be changed to "packages installed from an extracted sdist tarball do not install documentation", but I haven't tested that thoroughly enough yet.

[nejucomo]

I just created ​this change to setup.py which causes all of ./docs and some other useful files like README.txt to be installed in $PREFIX/share/doc/allmydata_tahoe when installed with pip.

Next I'd like to test that this is also the case when I run python ./setup.py install --single-version-externally-managed from an unpacked sdist.

If that also works, then I would propose to merge that branch.

There are a few sticking points in that branch:

• It replicates some information from MANIFEST.in. (I found while doing this that MANIFEST.in refers to non-existent paths, and that python ./setup.py sdist does not complain about these missing files, so it has accumulated some bitrot.)
• It may use the wrong path format for all platforms... I suspect it should use distutils.util.convert_path() somewhere.
• There's no test code. Testing build/install scaffolding is difficult!

[nejucomo]

Here's a demo of the ​1659-bundle-docs-in-sdist branch using pip to install from a git repository:

 $ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD
1659-bundle-docs-in-sdist
b0ca93b732ba47cdd14d9b70996525c7cc8c85a1


$ TMP="$(mktemp --tmpdir -d 1659-bundle-docs-in-sdist.demo-XXXXXX)"


$ virtualenv "$TMP"
Using real prefix '/usr/local/bin/../stow/Python-2.7.3-setuptools-0.6c11'
New python executable in /home/n/tmp/1659-bundle-docs-in-sdist.demo-I4S29A/bin/python
Installing setuptools...............done.
Installing pip.................done.


$ source "$TMP/bin/activate"
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ pip install twisted > $TMP/pip-install-twisted.log # necessary work around; see https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2032
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ pip install . > $TMP/pip-install-dot.log
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ cd "$VIRTUAL_ENV"
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ ls ./share/doc/allmydata-tahoe/
COPYING.GPL  COPYING.TGPPL.rst  CREDITS  docs  NEWS.rst  README.txt  relnotes.txt
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ find share/doc/ -type d
share/doc/
share/doc/zfec
share/doc/pyutil
share/doc/pycryptopp
share/doc/allmydata-tahoe
share/doc/allmydata-tahoe/docs
share/doc/allmydata-tahoe/docs/historical
share/doc/allmydata-tahoe/docs/frontends
share/doc/allmydata-tahoe/docs/proposed
share/doc/allmydata-tahoe/docs/specifications
share/doc/allmydata-tahoe/docs/specifications/backends
share/doc/allmydata-tahoe/docs/man
(1659-bundle-docs-in-sdist.demo-I4S29A)

[nejucomo]

Here is a demo which does not use pip nor virtualenv:

$ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD
1659-bundle-docs-in-sdist
b0ca93b732ba47cdd14d9b70996525c7cc8c85a1

$ TMP="$(mktemp --tmpdir -d lafs-install-no-virtual-env-or-pip.demo-XXXXXX)"

$ python ./setup.py sdist > "$TMP/sdist.log"
warning: no previously-included files matching '*~' found anywhere in distribution

$ tar -xf ./dist/allmydata-tahoe-1.10.0.post32.tar.gz -C /usr/local/src

$ cd /usr/local/src/allmydata-tahoe-1.10.0.post32/

$ python ./setup.py install --single-version-externally-managed --record "$TMP/install-record" --prefix='/usr/local' > "$TMP/install.log"
warning: no previously-included files matching '*~' found anywhere in distribution

$ find /usr/local/share/doc/allmydata-tahoe/ -type d
/usr/local/share/doc/allmydata-tahoe/
/usr/local/share/doc/allmydata-tahoe/docs
/usr/local/share/doc/allmydata-tahoe/docs/historical
/usr/local/share/doc/allmydata-tahoe/docs/frontends
/usr/local/share/doc/allmydata-tahoe/docs/proposed
/usr/local/share/doc/allmydata-tahoe/docs/specifications
/usr/local/share/doc/allmydata-tahoe/docs/specifications/backends
/usr/local/share/doc/allmydata-tahoe/docs/man

[zooko]

Replying to nejucomo:

• It may use the wrong path format for all platforms... I suspect it should use distutils.util.convert_path() somewhere.
• There's no test code. Testing build/install scaffolding is difficult!

We have a (kludgey) way to test such things. Here is the output on buildbot:

A test of ['python', '-c', "import glob, os, subprocess, sys; os.mkdir('egginstalldir'); tahoe_egg = glob.glob(os.path.join('dist', '*.egg'))[0]; sys.exit(subprocess.call(['easy_install', '-d', 'egginstalldir', tahoe_egg]))"]:

​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-egg/logs/stdio

A test of ['python', '-c', "import subprocess, sys;sys.exit(subprocess.call([sys.executable, 'setup.py', 'install', '--single-version-externally-managed', '--record=record.txt', '--prefix', 'prefixinstalldir']))"]:

​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-prefix/logs/stdio

In both cases, they are followed by executing the unit tests from within the installed location:

​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-egg

and

​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-prefixdir

So to test that the docs are installed correctly, I guess we could add another Buildbot BuildStep? which runs after the install and before the unit-tests-run-from-installed, which just asserts that the right files are in place.

Nathan: interested in contributing such a test?

[nejucomo]

Since I've already started a patch for this, I've assigned it to myself.

[daira]

What's the status of this ticket?