"make test" tests the installed version of allmydata, not the local sandbox version of allmydata

[zooko]

It seems like .pth paths get searched earlier than PYTHONPATH, so if you install allmydata and then change something in your local sandbox and then run "make test", it will test the installed version, not the local one that you changed.

[warner]

eek. I thought that .pth files had path-manipulation logic in them to make sure that everything which gets added is really inserted into the middle of sys.path, in the same place that the .pth file was found.

Hmm. I went to confirm this, and realized that I can't seem to find any .pth files on my system that contain code, and I can't seem to find a reference to '.pth' in the site.py files there.

[zooko]

This one bit me again, as I sleepily couldn't figure out why my new unit test wasn't failing as it should. When I deleted the installed version of allmydata, this was fixed.

One workaround to this which might not be a terrible idea in any case is to exclude test files from the installed files.

[warner]

I'd like to understand the reasons first.. what was the PYTHONPATH used for that test?

I'm also inclined to leave the test sub-package installed, since that allows users who have just installed tahoe (perhaps from some binary installer) to run the unit tests on their own. FWIW, Twisted does the same thing, and following their lead I did the same for Buildbot and Foolscap.

[zooko]

When tahoe isn't installed, then make test starts with:

PYTHONPATH="/Users/wonwinmcbrootles/playground/allmydata/tahoe/src::" \
 python -u "/usr/local/bin/trial" --rterrors   allmydata.test
Running 233 tests.

I'll investigate more later...

[zooko]

Okay here is the difference between sys.path when running unit tests when tahoe is installed vs. when it is not installed:

-    test_options ... sys.path: ['/Users/wonwinmcbrootles/playground/allmydata/tahoe', '/usr/local/stow/python-release25-maint/bin', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zfec-unknown-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/foolscap-0.1.6-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/simplejson-1.7.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyutil-1.3.2-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/z_base_32-0.9.14-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/Nevow-0.9.18-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyflakes-0.2.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/allmydata_tahoe-0.6.0_76-py2.5-macosx-10.3-i386.egg', '/Users/wonwinmcbrootles/playground/allmydata/tahoe/src', '/usr/local/stow/python-release25-maint/lib/python25.zip', '/usr/local/stow/python-release25-maint/lib/python2.5', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-darwin', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-tk', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-dynload', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages']
+    test_options ... sys.path: ['/Users/wonwinmcbrootles/playground/allmydata/tahoe', '/usr/local/stow/python-release25-maint/bin', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zfec-unknown-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/foolscap-0.1.6-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/simplejson-1.7.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyutil-1.3.2-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/z_base_32-0.9.14-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/Nevow-0.9.18-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyflakes-0.2.1-py2.5.egg', '/Users/wonwinmcbrootles/playground/allmydata/tahoe/src', '/usr/local/stow/python-release25-maint/lib/python25.zip', '/usr/local/stow/python-release25-maint/lib/python2.5', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-darwin', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-tk', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-dynload', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages']

So all of the installed .eggs from the Python site-packages dir, including the allmydata-tahoe .egg, are earlier on the path than $PWD/src is.

[zooko]

Bumping to v0.7.

[warner]

this is currently giving us false-positive test results on the Solaris buildslave, and possibly others, so I'm raising the priority.

One way to lower the priority would be to modify the test process and buildbot config to emit the version number of the code actually being tested (as extracted from the _trial_temp/twistd.log file) in the test step, so we could spot the discrepancy. At the moment, the buildbot is silently testing the wrong versions on platforms where tahoe was installed into a normal system path.

[warner]

from the following mailing list articles:

​http://www.eby-sarna.com/pipermail/peak/2006-june/002582.html ​http://mail.python.org/pipermail/distutils-sig/2006-July/006492.html

it looks like using eggs always results in the following import order:

• eggs on PYTHONPATH
• eggs in site-packages
• non-eggs on PYTHONPATH
• non-eggs in site-packages

and for us, the tahoe "egg in site-packages" is overriding the code-under-test "non-egg on PYTHONPATH".

It appears that the recommended solution might be to use "setup.py develop", to create an .egg (really a directory, rather than a .zip file) that contains symlinks into our source tree. Then put that .egg in a local directory on PYTHONPATH.

[zooko]

We're focussing on an imminent v0.7.0 which hopefully has Small Distributed Mutable Files and also a fix for bad SHA-256. So I'm bumping less urgent tickets to v0.7.1.

[zooko]

I uninstalled the allmydata-tahoe package from the system on nooxie, so now the buildslave on nooxie is testing its local checkout of the trunk source.

So while this is still a bug, it is no longer the cause of any build failures on nooxie.

[zooko]

Brian, do I understand correctly that the fix is for "make test" to do "./setup.py develop" to produce an egg which symlinks into the directory and put that egg on the PYTHONPATH?

[zooko]

Nowadays make build runs ./setup.py develop, so this might fix this bug. I haven't tested it yet, though.

[zooko]

Yes, this is fixed.

[zooko]

Hm. This is apparently not fixed. If you easy_install tahoe then make test tests that installed version instead of the one in ./support.

[zooko]

No, I checked and this seems to be fixed. If anyone else can reproduce the problem -- make test testing some other copy of the allmydata-tahoe source code instead of the source code in the current directory -- please let us know.

Also, I guess, if ./setup.py trial tests any allmydata package other than the one in the current directory, that too would mean this ticket should be re-opened.

[davidsarah]

See #1137 for a reoccurrence of this problem, in the test-from-egg and test-from-prefixdir buildbot steps.

Why do we think it was fixed for setup.py trial? I see no test that would fail if we were testing the wrong code. (test_path in test_runner.py​ checks that we run the same code by invoking bin/tahoe that we are currently testing, but if bin/tahoe and setup.py trial were running the same installed version -- which is entirely plausible because they both work by setting PYTHONPATH -- then we wouldn't detect that.)

[zooko]

Hm, test_path in test_runner.py​ detects whether starting from the allmydata directory that contains the allmydata/__init__.py file and going ../../bin/tahoe finds a working tahoe executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac:

 Zooko-Ofsimplegeos-MacBook-Pro:~$ python -i
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import allmydata
>>> import os
>>> os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(allmydata.__file__))), 'bin', 'tahoe')
'/Library/Python/2.6/site-packages/bin/tahoe'
>>> ^D
 Zooko-Ofsimplegeos-MacBook-Pro:~$ ls -al /Library/Python/2.6/site-packages/bin/tahoe
ls: /Library/Python/2.6/site-packages/bin/tahoe: No such file or directory
 Zooko-Ofsimplegeos-MacBook-Pro:~$ python -c 'import allmydata;print allmydata.__file__'
/Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg/allmydata/__init__.pyc

Ah, but in this case the test is skipped:

HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/trunk/xyz$ trial allmydata.test.test_runner.TheRightCode.test_path
allmydata.test.test_runner
  TheRightCode
    test_path ...                                                     [SKIPPED]

===============================================================================
[SKIPPED]
The bin/tahoe script isn't to be found in the expected location (/Library/Python/2.6/site-packages/bin/tahoe), and I don't want to test a 'tahoe' executable that I find somewhere else, in case it isn't the right executable for this version of Tahoe. Perhaps running 'setup.py build' again will help.

allmydata.test.test_runner.TheRightCode.test_path
-------------------------------------------------------------------------------
Ran 1 tests in 0.011s

PASSED (skips=1)

So this test does fail in (some) cases that it is running a test_runner.py from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm.

[davidsarah]

Replying to zooko:

Hm, test_path in test_runner.py​ detects whether starting from the allmydata directory that contains the allmydata/__init__.py file and going ../../bin/tahoe finds a working tahoe executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac:

[...]

Ah, but in this case the test is skipped:

HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/trunk/xyz$ trial allmydata.test.test_runner.TheRightCode.test_path
allmydata.test.test_runner
  TheRightCode
    test_path ...                                                     [SKIPPED]

Actually the test will always be skipped rather than failing because of this. It is skipped in the ​case in #1137, for example.

So this test does fail in (some) cases that it is running a test_runner.py from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, ...

I think running the tests on an installed version (as opposed to a version that is built at a prefixdir) isn't currently a supported thing to do; you have to be in a directory that contains at least a setuptools-*.egg, and presumably that also looks like a Tahoe-LAFS source distribution in other ways (the missing setuptools-*.egg is as far as I got). That it passes when you import an installed version instead of the one in src/allmydata is an accident; I'll open another ticket to support that case.

and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm.

The test_the_right_code test that I ​added on the ticket1074 branch should fail in this case, but an additional problem is that the version we're incorrectly testing might not have that test (or, we might fix a bug in that the test in future, and be confused by its difference in behaviour in the version we're actually testing).

Making a change in setuptools_trial would have the same problem because we might be using an installed version of that. Making a change in build_helpers/run_trial.py would work, though. I'm trying that approach now.

[zooko]

Hm, let's list the supported ways to run tests:

1. cd $SOURCEDIR && python setup.py test
2. cd $SOURCEDIR && trial allmydata.test
3. mkdir tempempty && cd tempempty && trial allmydata.test # should run unit tests of the installed source code, might skip some tests -- test_runner -- that depend on the test code figuring our where the 'tahoe' executable lives
4. mkdir -p egginstalldir && PYTHONPATH=egginstalldir:${PYTHONPATH} easy_install -d egginstalldir dist/*.egg && cd egginstalldir && PYTHONPATH=.:${PYTHONPATH} trial allmydata.test # should run unit tests of the code in this egginstalldir including test_runner. This is what install-to-egg and test-from-egg on the buildbot is trying to do.
5. python setup.py install --single-version-externally-managed --record=/dev/null --prefix=prefixinstalldir && cd prefixinstalldir && PYTHONPATH=lib/python2.6/site-packages:${PYTHONPATH} PATH=bin:${PATH} trial allmydata.test # should run unit tests of the code in this prefixinstalldir including test_runner. This is what install-to-prefixdir and test-from-prefixdir on the buildbot is trying to do

Is that it?

Note: I just tried each of these manually and here are my results:

1. works
2. does not work; It runs the unit tests of the installed version in /Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg instead of the current source tree, even if I set the PYTHONPATH before running trial.
3. works
4. works
5. does not work; It runs the unit tests of the installed version in /Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg instead of the current source tree, even if I set the PYTHONPATH before running trial.

[davidsarah]

Note that as part of #1296, all of the supported ways to run tests (including the Makefile targets) now directly or indirectly invoke bin/tahoe debug trial. The ways to run tests described in comment:26 that didn't work, still don't work, but they are no longer used on the buildbots and I don't think they should be considered supported.

It is still possible for --single-version-externally-managed installs (and other installs that copy code to site directories, such as the Twisted installers for Windows) to mess things up, although we try very hard to detect such cases. The only such cases I'm aware of are caused by #1258. Can we consider this ticket to be a duplicate of that one?

[zooko]

Let's document the "supported and unsupported ways to run tests" from comment:26. Perhaps in a new file named docs/tests.rst?

[davidsarah]

Replying to zooko:

Let's document the "supported and unsupported ways to run tests" from comment:26. Perhaps in a new file named docs/tests.rst?

Filed as #1439. I haven't seen any other cases than #1258 where we test the wrong code, so I'm marking this as a duplicate.