Opened at 2010-11-02T14:52:36Z

Closed at 2011-01-17T22:45:20Z

#1246 closed defect (duplicate)

figure out why "FreeStorm Win7-amd64-mingw py2.6" is red on the "test-with-fake-pkg" step

Reported by: zooko Owned by: zooko
Priority: major Milestone: soon
Component: packaging Version: 1.8.0
Keywords: buildbot setuptools test-needed Cc:
Launchpad Bug:

Description (last modified by davidsarah)

(Note that the two failures here have different causes. From comment:4 onwards, this ticket is about the second one.)

MM's buildslave "MM netbsd5 i386 warp" fails the test-from-egg step, like this: 

Traceback (most recent call last):
  File "/home/tahoe/buildslave/tahoe/netbsd5-i386/build/misc/build_helpers/run_trial.py", line 52, in <module>
    __import__(modulename)
ImportError: No module named allmydata.test

FS's buildslave "FreeStorm Win7-amd64-mingw py2.6" fails the test-with-fake-pkg step, like this: 

Traceback (most recent call last):
  File "c:\buildbot_tahoe\FreeStorm_Win7-amd64-mingw_py2.6\build\src\..\misc\build_helpers\run_trial.py", line 98, in <module>
    raise AssertionError(msg)
AssertionError: We seem to be testing the code at 'c:\\python26\\lib'
(according to the source filename 'c:\\python26\\lib\\site-packages\\allmydata\\test\\test_base62.pyc'),
but expected to be testing the code at 'c:\\buildbot_tahoe\\freestorm_win7-amd64-mingw_py2.6\\build'.
This script needs to be run from the source directory to be tested.

(In ​a subsequent build, this buildslave ​hung, and then in subsequent builds it was ​unable to start due to a process holding a handle to the build directory.)

Attachments (4)

buildmaster-search-parent-for-eggs.darcs.patch​ (79.1 KB) - added by davidsarah at 2010-11-04T01:16:25Z.
buildmaster bbsupport.py: have the test-from-egg step look for eggs in the egginstalldir as well as its parent. refs #1246
freestorm-test-with-fake-pkg-output.txt​ (6.2 KB) - added by davidsarah at 2010-11-05T01:00:40Z.
output of test-with-fake-pkg.py on freestorm's builder, with printing of sys.path before and after 'import pkg_resources' in run_trial.py
install-should-only-install-scripts.darcs.patch​ (19.3 KB) - added by davidsarah at 2010-11-06T00:29:08Z.
setup.py, setup.cfg, bin/tahoe-script.template: make it so that 'python setup.py install' just does a build and installs a script pointing to that build. refs #1246
misc-build_helper-test-1246.darcs.patch​ (12.6 KB) - added by zooko at 2011-01-01T23:55:43Z.

Download all attachments as: .zip

Change History (27)

comment:1 Changed at 2010-11-03T18:42:34Z by midnightmagic

For the warp client, there isn't an allmydata egg except inside the egginstalldir itself; therefore the issue seems to be:

Code highlighting: 

import glob,os,subprocess,sys
trial=os.path.join(os.getcwd(), 'misc', 'build_helpers', 'run_trial.py')
os.chdir('.')

Right in here...

If I change the above os.chdir('.') to os.chdir('egginstalldir') then the test proceeds as normal. Since this is ostensibly a test from the egginstalldir location, I fail to see why it should be globbing the eggs from the enclosing directory prior to changing directory into the egginstalldir and then running the trial from that directory after overriding PYTHONPATH.

So I guess I am failing to pierce the meaning behind the way we are prepending the eggs' realpath() to PYTHONPATH from pre- to post- chdir(). 

eggs=[os.path.realpath(egg) for egg in
      glob.glob('*.egg')]
testsuite='allmydata.test'
os.chdir('egginstalldir')
os.environ['PATH']=os.getcwd()+os.pathsep+os.environ['PATH']
os.environ['PYTHONPATH']=os.pathsep.join(eggs)+os.pathsep+os.environ.get('PYTHONPATH','')
sys.exit(subprocess.call([sys.executable, trial, '--rterror', testsuite], env=os.environ))

Changed at 2010-11-04T01:16:25Z by davidsarah

buildmaster bbsupport.py: have the test-from-egg step look for eggs in the egginstalldir as well as its parent. refs #1246

comment:2 follow-up: Changed at 2010-11-04T01:39:01Z by davidsarah

  • Owner changed from midnightmagic to zooko

zooko: please apply buildmaster-search-parent-for-eggs.darcs.patch​ and restart the buildmaster.

comment:3 in reply to: ↑ 2 ; follow-up: Changed at 2010-11-04T06:34:34Z by zooko

  • Owner changed from zooko to freestorm
  • Summary changed from figure out why MM's and FS's buildslaves are red on some steps to figure out why "FreeStorm Win7-amd64-mingw py2.6" is red on the "test-with-fake-pkg" step

Replying to davidsarah:

zooko: please apply buildmaster-search-parent-for-eggs.darcs.patch​ and restart the buildmaster.

Done:

​http://tahoe-lafs.org/buildbot/builders/MM%20netbsd5%20i386%20warp/builds/135

It looks like MM's buildslave is all fixed up now. FreeStorm's Win67-amd64-mingw-py2.6 is not, though:

​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/154/steps/test-with-fake-pkg/logs/stdio

comment:4 in reply to: ↑ 3 ; follow-up: Changed at 2010-11-05T00:59:15Z by davidsarah

Replying to zooko:

FreeStorm's Win67-amd64-mingw-py2.6 is not [fixed], though:

​http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/154/steps/test-with-fake-pkg/logs/stdio

The Requirements: ['allmydata-tahoe==1.8.0-r4803', ...] line indicates what misc/build_helpers/run_trial.py set the __requires___ variable to. That is supposed to tell pkg_resources to insert entries on sys.path so that only modules from exactly that version of Tahoe-LAFS will be imported. But in fact the installed version (which cannot be 1.8.0-r4803 because that number increases with each build) is being run.

I asked freestorm to modify run_trial.py to print out sys.path just before and just after importing pkg_resources. The results show no specific version of the allmydata-tahoe package on either of these paths. In particular, the src directory of the Tahoe-LAFS distribution is not on the "before" path, as I would have expected. (That is the current directory when executing run_trial.py from misc/build_helpers/test-with-fake-pkg.py.)

The code in pkg_resources that inserts path entries is difficult to follow, but I think that it only reorders entries and adds .egg files from directories that are already on the sys.path, which would explain why it's not able to find the correct version of allymydata-tahoe. Perhaps it would help to explicitly add the src directory to the PYTHONPATH variable in test-with-fake-pkg.py; I'll ask freestorm to try that.

Changed at 2010-11-05T01:00:40Z by davidsarah

output of test-with-fake-pkg.py on freestorm's builder, with printing of sys.path before and after 'import pkg_resources' in run_trial.py

comment:5 in reply to: ↑ 4 ; follow-up: Changed at 2010-11-05T01:40:49Z by davidsarah

Replying to davidsarah:

I asked freestorm to modify run_trial.py to print out sys.path just before and just after importing pkg_resources. The results show no specific version of the allmydata-tahoe package on either of these paths. In particular, the src directory of the Tahoe-LAFS distribution is not on the "before" path, as I would have expected.

Sorry, I was mistaken. It is on ​line 24 "before" and line 68 "after". The reason why the installed Tahoe is being picked up instead is that c:\python26\lib\site-packages is earlier than c:\buildbot_tahoe_0.8.2\freestorm_win7-amd64-mingw_py2.6\build\src on the "after" path, even though it is later on the "before" path. So importing pkg_resources incorrectly changed the order of these two entries.

Last edited at 2010-11-05T23:24:02Z by davidsarah (previous) (diff)

comment:6 in reply to: ↑ 5 ; follow-ups: Changed at 2010-11-05T04:54:51Z by davidsarah

Replying to davidsarah:

The reason why the installed Tahoe is being picked up instead is that c:\\python26\\lib\\site-packages is earlier than c:\\buildbot_tahoe_0.8.2\\freestorm_win7-amd64-mingw_py2.6\\build\\src on the "after" path, even though it is later on the "before" path. So importing pkg_resources incorrectly changed the order of these two entries.

Freestorm and I had a long debugging session and worked out what was causing this. The buildslave has three distributions that are installed directly in C:\Python26\lib\site-packages: allmydata_tahoe-1.8.0c1_r4687-py2.6, pywin32-214-py2.6.egg-info and Twisted-10.1.0-py2.6. Each is installed as an .egg-info directory and source files with the same root (e.g. Twisted-10.1.0-py2.6 and twisted both in site-packages).

That meant that when pkg_resources tried to add an entry for Twisted (10.1.0) to sys.path, it added the location C:\Python26\lib\site-packages, which effectively also added Tahoe 1.8.0c1-r4687 to sys.path with the same priority. The correct Tahoe 1.8.0-r4803 was also later on the path, but pkg_resources makes no attempt to order the requirements specified in __requires__ relative to each other.

.egg-info installations directly into site-packages are apparently quite common :-( On my machine I have Tahoe (several .egg-infos for different versions), pywin32, Twisted 9.0.0, pycairo, pyflakes, and zetuptoolz installed in that way. The pywin32 and Twisted .exe installers apparently do this by default.

According to ​this stackoverflow answer, ".egg-info directories get only created if --single-version-externally-managed was used to install the egg". My conclusion from this is that we should never use --single-version-externally-managed to install Tahoe into any directory where other distributions might also be installed.

Anyway, Freestorm deleted the allmydata and allmydata_tahoe-1.8.0c1_r4687-py2.6.egg-info directories from site-packages (after taking a copy), and that ​made test-with-fake-pkg go green.

comment:7 Changed at 2010-11-05T04:59:26Z by davidsarah

  • Component changed from dev-infrastructure to packaging
  • Keywords setuptools added

comment:8 in reply to: ↑ 6 ; follow-up: Changed at 2010-11-05T13:24:13Z by zooko

So if I understand correctly, the problem you are focussing on is that sometimes distributions get installed by copying their modules directly into some directory which is on sys.path. This makes it so that import $MODULENAME will subsequently work but it is a problem, if I understand you correctly, if more than one distribution is installed into the same directory because then there is no way to prioritize one of the distributions without thereby also prioritizing all the other distributions in the same directory. By "prioritize" here I guess I mean move it closer to the front of sys.path. Other methods of manipulating its precedence before importing would be moving it closer to the end of the sys.path or removing the directory from the sys.path entirely. None of these can be done on only a single distribution if more than one distribution has been installed "directly" (by dint of copying its modules into place) in a single directory.

One thing to note is that the standard distutils, without setuptools/zetuptoolz/distribute, always installs things in this way! If you're using the standard distutils, the only way to avoid doing this would be to create a new empty directory and install into that (with --prefix) and then add that to the sys.path. You would have to do that for every distribution that you install.

Okay, so now I can appreciate David-Sarah's conclusion: "We should never use --single-version-externally-managed to install Tahoe into any directory where other distributions might also be installed.". That makes sense. We already do avoid that, as far as I know. For example, ​quickstart.html doesn't advise the user to do that.

comment:9 in reply to: ↑ 8 ; follow-up: Changed at 2010-11-05T17:25:44Z by davidsarah

Replying to zooko:

So if I understand correctly, the problem you are focussing on is that sometimes distributions get installed by copying their modules directly into some directory which is on sys.path. This makes it so that import $MODULENAME will subsequently work but it is a problem, if I understand you correctly, if more than one distribution is installed into the same directory because then there is no way to prioritize one of the distributions without thereby also prioritizing all the other distributions in the same directory. By "prioritize" here I guess I mean move it closer to the front of sys.path. Other methods of manipulating its precedence before importing would be moving it closer to the end of the sys.path or removing the directory from the sys.path entirely. None of these can be done on only a single distribution if more than one distribution has been installed "directly" (by dint of copying its modules into place) in a single directory.

Exactly.

One thing to note is that the standard distutils, without setuptools/zetuptoolz/distribute, always installs things in this way!

However, the problem in this ticket only applies to Tahoe-LAFS dependencies (direct or indirect, including the allmydata-tahoe package itself) that are installed in this way.

This could be worked around in several ways:

a) change zetuptoolz to order "development" dists (directories containing Python code and an .egg-info subdirectory) after non-development dists when satisfying __requires__ lists. Note: the pkg_resources code involved is complicated and I don't understand it.

b) change zetuptoolz so that it does not view development dists as satisfying a dependency. That would cause a non-development dist to be used automatically instead. (pywin32 is a potential spanner in the works because it can't currently be installed as an egg, but maybe #142 could be fixed.) Fixing #1233 is important here, because otherwise you would have regressions where a package is already installed but only as a development dist, and building a new version of it fails because it can't be compiled.

c) change the wrapper scripts not to use the __requires__ mechanism and to put the dependencies directly at the start of sys.path itself. This would still have to avoid viewing development dists as satisfying a dependency, or else it would run into the same problem.

[...]

Okay, so now I can appreciate David-Sarah's conclusion: "We should never use --single-version-externally-managed to install Tahoe into any directory where other distributions might also be installed.". That makes sense. We already do avoid that, as far as I know. For example, ​quickstart.html doesn't advise the user to do that.

As far as I can see, nothing in our docs advises against installing Tahoe using setup.py install or easy_install. That's a standard way of installing Python packages.

Besides, there's no good reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason.

comment:10 in reply to: ↑ 9 ; follow-ups: Changed at 2010-11-05T17:34:45Z by davidsarah

Replying to davidsarah:

As far as I can see, nothing in our docs advises against installing Tahoe using setup.py install or easy_install. That's a standard way of installing Python packages.

Besides, there's no good reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason.

Hmm, one way to make this work is for setup.py install (and therefore easy_install) to actually do a build rather than an install, but also install a tahoe wrapper script for that build to the global scripts directory.

comment:11 in reply to: ↑ 10 ; follow-up: Changed at 2010-11-05T17:51:41Z by davidsarah

Replying to davidsarah:

Replying to davidsarah:

As far as I can see, nothing in our docs advises against installing Tahoe using setup.py install or easy_install. That's a standard way of installing Python packages.

Besides, there's no good reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason.

Hmm, one way to make this work is for setup.py install (and therefore easy_install) to actually do a build rather than an install, but also install a tahoe wrapper script for that build to the global scripts directory.

Maybe this is as simple as putting "install = darcsver --count-all-patches develop --prefix=support make_executable build install_scripts" in place of the existing install alias in Tahoe's setup.cfg​.

comment:12 in reply to: ↑ 11 Changed at 2010-11-05T23:21:26Z by davidsarah

Replying to davidsarah:

Maybe this is as simple as putting "install = darcsver --count-all-patches develop --prefix=support make_executable build install_scripts" in place of the existing install alias in Tahoe's setup.cfg​.

Apparently not. That does cause python setup.py install to build Tahoe successfully and generate C:\Python26\Scripts\tahoe.pyscript, and the resulting script has a correct __requires__ line. Nevertheless it fails (on this machine) with the error 

>tahoe --version
Traceback (most recent call last):
  File "C:\Python26\Scripts\tahoe.pyscript", line 6, in <module>
    from pkg_resources import load_entry_point
  File "C:\Python26\lib\site-packages\setuptools-0.6c16dev.egg\pkg_resources.py", line 2607, in <module>
    parse_requirements(__requires__), Environment()
  File "C:\Python26\lib\site-packages\setuptools-0.6c16dev.egg\pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: pycryptopp>=0.5.20

This is understandable because pycryptopp>=0.5.20 is not globally installed, the support dir of the Tahoe build is not on the PYTHONPATH, and C:\Python26\Scripts\tahoe.pyscript does not put it onto the PYTHONPATH (since it is a zetuptoolz-generated script rather than the one from Tahoe's bin directory). Also note that it is using pkg_resources from setuptools-0.6c16dev, which is the globally installed version.

Getting setup.py install to work as I want it to is possible, but might have to wait for Brian's "unsuck" branch, since it's probably not a good idea to make the zetuptoolz-generated script as complicated as the 'bin/tahoe' one.

Changed at 2010-11-06T00:29:08Z by davidsarah

setup.py, setup.cfg, bin/tahoe-script.template: make it so that 'python setup.py install' just does a build and installs a script pointing to that build. refs #1246

comment:13 in reply to: ↑ 10 ; follow-up: Changed at 2010-11-06T00:36:01Z by davidsarah

  • Keywords review-needed test-needed added

Replying to davidsarah:

Hmm, one way to make this work is for setup.py install (and therefore easy_install) to actually do a build rather than an install, but also install a tahoe wrapper script for that build to the global scripts directory.

install-should-only-install-scripts.darcs.patch​ implements that. If we agree that it's a good idea, we would need a buildbot step to test it.

comment:14 in reply to: ↑ 13 Changed at 2010-11-06T01:23:25Z by davidsarah

Replying to davidsarah:

Replying to davidsarah:

Hmm, one way to make this work is for setup.py install (and therefore easy_install) to actually do a build rather than an install, but also install a tahoe wrapper script for that build to the global scripts directory.

install-should-only-install-scripts.darcs.patch​ implements that. If we agree that it's a good idea, we would need a buildbot step to test it.

... and reconsider how to implement install-to-prefix, which this patch breaks.

comment:15 in reply to: ↑ 6 Changed at 2011-01-01T09:14:25Z by davidsarah

Replying to davidsarah:

According to ​this stackoverflow answer, ".egg-info directories get only created if --single-version-externally-managed was used to install the egg". My conclusion from this is that we should never use --single-version-externally-managed to install Tahoe into any directory where other distributions might also be installed.

Unfortunately, that's probably not sufficient (although it is necessary). The bug requires that at least one Tahoe dependency be installed as an .egg-info, but the other distribution in the same directory that gets imported by mistake does not itself have to have an .egg-info directory.

So, we should never use --single-version-externally-managed or equivalent to install Tahoe or any of its dependencies into any directory where other distributions might also be installed. But we can't control how users install, say, Twisted or pywin32 (which have installers that are equivalent to using --single-version-externally-managed).

Changed at 2011-01-01T23:55:43Z by zooko

comment:16 Changed at 2011-01-02T00:04:46Z by zooko

attachment:misc-build_helper-test-1246.darcs.patch​ is my attempt to exercise this problem. However, I don't think this attempt succeeds. When I run python misc/build_helpers/test-dists-in-shared-directory.py, I get this result: 

Requirements: ['allmydata-tahoe==1.8.1-r4897', 'setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface', 'Twisted == 10.1.0', 'foolscap[secure_connections] >= 0.6.0', 'Nevow >= 0.6.0', 'pycrypto == 2.0.1, == 2.1, >= 2.3', 'pyasn1 >= 0.0.8a', 'pycryptopp >= 0.5.20', 'mock']
Inserting egg on sys.path: '/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg'
buildtest.test_dists_in_shared_directory
  T
    test_version ...                                                    [ERROR]

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/Users/zooko/playground/tahoe-lafs/trunk/src/buildtest/test_dists_in_shared_directory.py", line 8, in test_version
    pkg_resources.require('pycryptopp >= 0.5.14')
  File "/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg/pkg_resources.py", line 676, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg/pkg_resources.py", line 575, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: pycryptopp>=0.5.14

buildtest.test_dists_in_shared_directory.T.test_version
-------------------------------------------------------------------------------
Ran 1 tests in 0.045s

FAILED (errors=1)

Maybe this means that the current zetuptoolz-based build system passes this test? Maybe the test that we want has two versions of the package, both of which are of a sufficiently high version number, and we are trying to show that the build system mistakenly imports one when pkg_resources reports that it uses the other.

Note that this test works only if you don't have a sufficient version of pycryptopp installed in your system. If there is already a sufficient version of pycryptopp, then you'll get a skip.

Whoops, in fact you get some sort of failure-to-run trial: 

Traceback (most recent call last):
  File "/Users/zooko/playground/tahoe-lafs/trunk/src/../misc/build_helpers/run_trial.py", line 52, in <module>
    __import__(modulename)
ImportError: No module named test_dists_in_shared_directory

comment:17 follow-up: Changed at 2011-01-02T00:19:53Z by davidsarah

In at least some cases (#1258), setup.py finds the correct code but bin/tahoe does not. Perhaps that difference is involved in the misc-build_helper-test-1246.darcs.patch​ test's failure to reproduce the problem (since the test invokes run_trial.py which invokes setup.py).

comment:18 in reply to: ↑ 17 Changed at 2011-01-02T01:22:11Z by davidsarah

Replying to davidsarah:

In at least some cases (#1258), setup.py finds the correct code but bin/tahoe does not. Perhaps that difference is involved in the misc-build_helper-test-1246.darcs.patch​ test's failure to reproduce the problem (since the test invokes run_trial.py which invokes setup.py).

Here is a untried test that imports pycryptopp in a subprocess, similar to test_import_from_repl in ticket:1258:add-test-import-in-repl.darcs.patch​. Unfortunately it is complicated and depends on more than we want to depend on for this test, but let's see whether it helps to reproduce the problem. 

from twisted.trial import unittest
import subprocess, sys

# TODO: move these to common or common_util
from allmydata.test.test_runner import bintahoe, SkipMixin

class T(unittest.TestCase, SkipMixin):
    def test_version(self):
        # based on test_import_in_repl
        self.skip_if_cannot_run_bintahoe()

        p = subprocess.Popen([sys.executable, bintahoe, "debug", "repl"],
                             stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
        (out, err) = p.communicate("import pycryptopp; "+
                                   "print pycryptopp.__version__")

        self.failUnlessEqual(p.returncode, 0)
        self.failIfIn("Traceback", err)
        first = out.splitlines()[0]
        self.failUnless(first.startswith('>>> '))
        pycryptopp_version = first[4:]

        # If you tried to import 0.5.13 then the repl would have gotten
        # an exception and printed a Traceback to stderr, causing a
        # failure above. If there is already a pycryptopp installed in
        # the system then the import might have gotten that, instead of
        # the fake 99.0.0 that we intended. In that case, we can't easily
        # tell whether we got this far because we correctly skipped over
        # the insufficient 0.5.13 and imported the sufficient 99.0.0,
        # or because the system-provided pycryptopp was found before
        # the insufficient 0.5.13 was found, so this test can't draw a
        # conclusion about whether our import machinery is working
        # right.
        if pycryptopp_version != '99.0.0':
            raise unittest.SkipTest(
"""We can't tell if this worked because this system has a different
version of pycryptopp (%r) already installed. See comment in
misc/build_helpers/test_dists_in_shared_directory.py .
""" % (pycryptopp_version,))

[Edited to fix an import error.]

Last edited at 2011-01-02T01:31:29Z by davidsarah (previous) (diff)

comment:19 Changed at 2011-01-02T02:26:33Z by davidsarah

The test in comment:18 does reproduce the error for me: 

$ python misc/build_helpers/test-dists-in-shared-directory.py
Requirements: ['allmydata-tahoe==1.8.1-r4913', 'setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interf
ace', 'Twisted == 10.2.0', 'foolscap[secure_connections] >= 0.6.0', 'Nevow >= 0.6.0', 'pycrypto == 2.0.1, == 2.1, >= 2.3
', 'pyasn1 >= 0.0.8a', 'pycryptopp >= 0.5.20', 'mock']
Inserting egg on sys.path: 'd:\\tahoe\\tahoe-clean\\setuptools-0.6c16dev3.egg'
Traceback (most recent call last):
  File "d:\tahoe\tahoe-clean\src\..\misc\build_helpers\run_trial.py", line 52, in <module>
    __import__(modulename)
  File "d:\tahoe\tahoe-clean\src\buildtest\test_dists_in_shared_directory.py", line 7, in <module>
    from allmydata.test.test_runner import bintahoe, SkipMixin
  File "d:\tahoe\tahoe-clean\src\allmydata\test\__init__.py", line 31, in <module>
    initialize()
  File "d:\tahoe\tahoe-clean\src\allmydata\windows\fixups.py", line 15, in initialize
    from allmydata.util.encodingutil import canonical_encoding
  File "d:\tahoe\tahoe-clean\src\allmydata\util\encodingutil.py", line 13, in <module>
    from allmydata.util.fileutil import abspath_expanduser_unicode
  File "d:\tahoe\tahoe-clean\src\allmydata\util\fileutil.py", line 9, in <module>
    from pycryptopp.cipher.aes import AES
  File "d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages1\pycryptopp\__init__.py", line 1, in <module>
    raise Exception('Aha I caught you trying to import me. I am a fake pycryptopp 0.5.13 package and you should be require a newer version than me.')
Exception: Aha I caught you trying to import me. I am a fake pycryptopp 0.5.13 package and you should be require a newer version than me.

It still needs work, though. If I delete the temp directory and the code that creates the fake pycryptopp 0.5.13, I get a spurious failure: 

...
  File "d:\tahoe\tahoe-clean\src\allmydata\util\fileutil.py", line 9, in <module>
    from pycryptopp.cipher.aes import AES
ImportError: No module named cipher.aes

That is, the repl indirectly attempts to import pycryptopp.cipher.aes. Working around that by removing the dependency on AES from fileutil.py, I get a skip: 

[SKIPPED]: buildtest.test_dists_in_shared_directory.T.test_version

We can't tell if this worked because this system has a different
version of pycryptopp ('0.5.25-r757') already installed. See comment in
misc/build_helpers/test_dists_in_shared_directory.py .

(0.5.25-r757 is the version of pycryptopp that normally gets used when running Tahoe on this system. It is not installed in site-packages.)

The reason why this happens is that the bin/tahoe script adds the support/{Lib,lib/pythonx.x}/site-packages directory to PYTHONPATH before the entries that were inherited from test-dists-in-shared-directory.py. Therefore, it finds the real pycryptopp before the fake pycryptopp 99.0.0. I confirmed that by modifying the test to print PYTHONPATH in the repl, which gives (one entry on each line for clarity): 

d:\tahoe\tahoe-clean\support\Lib\site-packages;
;
d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages1;
d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages2;
d:\tahoe\tahoe-clean\darcsver-1.7.1.egg;
d:\tahoe\tahoe-clean\foolscap-0.6.0-py2.6.egg;
d:\tahoe\tahoe-clean\setuptools-0.6c16dev3.egg;
d:\tahoe\tahoe-clean\setuptools_darcs-1.2.11-py2.6.egg;
d:\tahoe\tahoe-clean\setuptools_trial-0.5.9.egg;

(Ignore the foolscap entry, that's a workaround for foolscap 0.6.0 not being found on my system.)

comment:20 follow-up: Changed at 2011-01-02T02:28:38Z by davidsarah

My conclusion at this point is that we need to know why the problem is only occurring in subprocesses, before we can write a reliable test.

comment:21 Changed at 2011-01-14T23:30:20Z by zooko

  • Keywords review-needed removed
  • Owner changed from freestorm to zooko
  • Status changed from new to assigned

Removing review-needed. I will try to write a test.

comment:22 Changed at 2011-01-17T05:59:01Z by david-sarah@…

In e52d97a917364cb0:

Undo the temporary hack to check the foolscap version. refs #1246

comment:23 in reply to: ↑ 20 Changed at 2011-01-17T22:45:20Z by davidsarah

  • Description modified (diff)
  • Resolution set to duplicate
  • Status changed from assigned to closed

Replying to davidsarah:

My conclusion at this point is that we need to know why the problem is only occurring in subprocesses, before we can write a reliable test.

This is due to a different order for entries in __requires__. With help from abadger1999 (Toshio Kuratomi), we worked out a reliable test, using only three dists (two versions of one package and one of another), that will be independent of any globally installed packages. I'll prepare a patch for that.

I'm now sure that this is the same issue as #1258, so I'm marking it as a duplicate (but all of the discussion in this ticket from comment:4 on is still relevant).

Note: See TracTickets for help on using tickets.