#668 closed defect (fixed)

easy_install allmydata-tahoe is broken

Reported by: francois Owned by: zooko
Priority: major Milestone: 1.8.2
Component: packaging Version: 1.3.0
Keywords: setuptools install ubuntu debian test-needed Cc: francois@…
Launchpad Bug:

Description

Trying to install Tahoe 1.3.0 with easy_install fails with this message.

$ easy_install-2.5 --quiet --prefix $DIR allmydata-tahoe==1.3.0

The required version of setuptools (>=0.6c12dev) is not available, and
can't be installed while this script is running. Please install
 a more recent version first, using 'easy_install -U setuptools'.

(Currently using setuptools 0.6c8 (/usr/lib/python2.5/site-packages))
error: Setup script exited with 2

This was tested on Ubuntu intrepid and Debian lenny machines using Python 2.5.

However, installation from the tarball (python setup.py install) works fine.

Change History (31)

comment:1 Changed at 2009-03-27T11:32:47Z by zooko

  • Milestone changed from undecided to 1.3.1

That's because setuptools >= 0.6c12dev is actually zetuptoolz -- our own toothpick of setuptools: http://allmydata.org/trac/zetuptoolz

I hadn't realized that this breaks easy_install. I would like to fix this ASAP, but I'm not sure how.

comment:2 Changed at 2009-04-07T20:24:30Z by zooko

  • Milestone changed from 1.4.0 to 1.4.1

Okay, I think I know how to fix this (just by renaming our version of setuptools to "zetuptoolz", but I don't understand why the current unit tests (such as this one: http://allmydata.org/buildbot/builders/Shawn%20intrepid%20amd64/builds/127/steps/install-egg/logs/stdio ) don't catch this bug, and I don't want to change the build/install code just before a release. So, tahoe 1.4.0 will ship with this bug and I'll move this ticket into the 1.4.1 Milestone.

comment:3 Changed at 2009-06-30T19:20:09Z by zooko

  • Owner changed from somebody to zooko
  • Status changed from new to assigned

comment:5 Changed at 2009-07-04T21:05:18Z by ndurner

From IRC:

(20:58:05) ndurner: Hmpf, my build instructions for Windows don't work anymore
(22:42:48) zooko: I think your report should go into ticket #668.
E:\allmydata-tahoe-1.4.0>python setup.py build
Not found: tahoe-deps
Not found: ../tahoe-deps
Traceback (most recent call last):
  File "setup.py", line 346, in <module>
    zip_safe=False, # We prefer unzipped for easier access.
  File "C:\Python25\lib\distutils\core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l
ine 219, in __init__
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l
ine 243, in fetch_build_eggs
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin
e 522, in resolve
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin
e 758, in best_match
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin
e 770, in obtain
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l
ine 286, in fetch_build_egg
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 452, in easy_install
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 482, in install_item
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 661, in install_eggs
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 936, in build_and_install
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 927, in run_setup
distutils.errors.DistutilsError: Setup script exited with error: Python was buil
t with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

comment:6 Changed at 2009-07-11T23:17:42Z by zooko

#721 was a duplicate of this (with useful information in the bug report).

comment:7 follow-up: Changed at 2009-07-15T22:54:52Z by zooko

  • Milestone changed from 1.5.0 to eventually

Sigh. I hate to allow this issue to remain unfixed, but there really isn't a good reason to hold up the TahoeLAFS v1.5 release on it. The official install instructions http://allmydata.org/source/tahoe/trunk/docs/install.html currently work on all supported platforms as far as I know, and all of the "flunkOnFailure" building/packaging buildsteps on all of the SupportedBuilders? are passing.

Therefore, by invoking the definition of "Supported" to exclude everyone for whom this system does not currently work, I hereby bump this ticket out of the v1.5 Milestone. :-)

comment:8 in reply to: ↑ 7 Changed at 2009-09-12T23:22:51Z by bewst

Replying to zooko:

The official install instructions http://allmydata.org/source/tahoe/trunk/docs/install.html currently work on all supported platforms as far as I know,

Unfortunately, it's failing for me on intrepid with an apparently-related symptom:

$ python setup.py build
running darcsver
setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and src/allmydata/_version.py already exists, so leaving it alone.
running develop
Checking .pth file support in support/lib/python2.5/site-packages
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 2] No such file or directory: 'support/lib/python2.5/site-packages/test-easy-install-8824.pth'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    support/lib/python2.5/site-packages

This directory does not currently exist.  Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).

Therefore, by invoking the definition of "Supported" to exclude everyone for whom this system does not currently work, I hereby bump this ticket out of the v1.5 Milestone. :-)

comment:9 Changed at 2009-09-14T01:51:57Z by zooko

Thank you for your bug report. Your problem is related to but not the same as this one. I've opened a new ticket for your issue -- #803 (python ./setup.py install -- can't create or remove files in install directory).

comment:10 Changed at 2009-09-24T19:24:57Z by cgalvan

  • Owner changed from zooko to cgalvan
  • Status changed from assigned to new

comment:11 Changed at 2009-09-24T19:57:01Z by cgalvan

[14:40] <RedG3> zooko: I switched to distribute and ran the test suite with tahoe and everything seems good to me :) I also did an easy_install of tahoe with distribute and it worked, but it installed setuptools since it's explicitly listed in the install_requires [14:41] <RedG3> I would like to test it on some other platforms, and if it works out I think we can safely change the install_requires to distribute if it is ok with you.

comment:12 Changed at 2009-09-24T22:42:31Z by zooko

Replacing zetuptoolz with Distribute, as Chris is currently doing, might effect #709, #717, #756, #763, #799, and possibly quite a few others. :-)

comment:13 Changed at 2010-03-25T05:04:48Z by davidsarah

  • Keywords setuptools install ubuntu debian added

setuptools delenda est.

comment:14 Changed at 2010-04-12T17:24:48Z by zooko

  • Milestone changed from eventually to 1.7.0
  • Owner changed from cgalvan to zooko
  • Status changed from new to assigned

My plan is to try replacing setuptools with distribute as well as submitting a couple of patches to distribute (http://bitbucket.org/tarek/distribute/issue/136/respect-the-pythonpath and http://bitbucket.org/tarek/distribute/issue/46/be-more-like-distutils-with-regard-to ).

comment:15 Changed at 2010-06-16T19:25:25Z by zooko

  • Milestone changed from 1.7.0 to soon

comment:16 Changed at 2011-01-16T10:20:12Z by zooko

I'm very happy to report that since I updated pycryptopp to v0.5.28, that now sudo easy_install allmydata-tahoe works on my Mac OS X laptop even if I have none of the dependencies pre-installed [*].

Now if only we had an automated test which would go red if sudo easy_install allmydata-tahoe were to ever stop working...

(Probably it already worked before I updated pycryptopp if you were using easy_install as provided by setuptools, but if your easy_install is provided by distribute then you need the workaround in pycryptopp-0.5.28 for this distribute bug.)

([*] Except the pycrypto dependency doesn't work because http://pycrypto.org is unreachable (as it has been for days) but the PyPI record at http://pypi.python.org/pypi/pycrypto/2.3 shows http://www.pycrypto.org/files/pycrypto-2.3.tar.gz as the "Download URL" so easy_install always tries to download that file if it needs pycrypto and then aborts the Tahoe-LAFS build when it can't connect to pycrypto.org. The only work-around I can come up with for this at the moment is to add --find-links=http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-sdists/ to your easy_install command-line.)

comment:17 follow-up: Changed at 2011-01-17T02:24:51Z by zooko

Hooray! It works. I had to update pycryptopp and zfec to avoid build-dependencies, and someone (presumably dlitz) had to fix http://pycrypto.org to work again, but now easy_install allmydata-tahoe works! :-)

comment:18 Changed at 2011-01-17T02:26:50Z by zooko

  • Keywords test-needed added

Now if only we had some sort of regression test of this functionality then I would be happy to close this ticket...

comment:19 in reply to: ↑ 17 ; follow-ups: Changed at 2011-01-17T05:38:56Z by davidsarah

Replying to zooko:

Hooray! It works. I had to update pycryptopp and zfec to avoid build-dependencies, and someone (presumably dlitz) had to fix http://pycrypto.org to work again, but now easy_install allmydata-tahoe works! :-)

Be careful: if you use easy_install allmydata-tahoe to install into a directory where other Python packages that are Tahoe dependencies are installed, and then later try to run a newer version from a build using the newer version's bin/tahoe, then you may get an error because the older version is found by mistake. This shouldn't happen, but it can because of #1246 #1258.

(setup.py install has the same problem.)

I therefore think that we should be recommending against installing Tahoe to any site directories (which is where easy_install will put it by default). It should be safe to install it to a fresh directory using easy_install allmydata-tahoe --prefix=<some fresh directory>.

Also note that easy_install is some random version of setuptools or distribute, so who knows what bugs it has?

Last edited at 2011-01-18T09:06:04Z by davidsarah (previous) (diff)

comment:20 Changed at 2011-01-17T06:31:05Z by zooko

  • Milestone changed from soon to 1.8.2

Oh, we also need to commit a fix to #1054 (ImportError?: No module named setuptools_darcs.setuptools_darcs) to trunk before this will work.

comment:21 Changed at 2011-01-17T06:31:34Z by zooko

  • Keywords review-needed added

comment:22 in reply to: ↑ 19 ; follow-up: Changed at 2011-01-17T06:38:24Z by zooko

I agree that we shouldn't recommend easy_install allmydata-tahoe as a way to set up Tahoe-LAFS, for the reasons you gave. Also because there are other improvements that we've implemented in our primary, supported method of setting up--quickstart.html--such as #1233 (prefer binary eggs which satisfy dependencies over source eggs which are even newer).

However, easy_install (and its cousin pip, which I hope also works now) are very popular, so I'm happy if Tahoe-LAFS is compatible with them.

In particular, this means that if there were another Python project which depended on Tahoe-LAFS, that project could mark Tahoe-LAFS as a requirement and then builds/installs of that project would automatically set up Tahoe-LAFS and all of its dependencies.

Last edited at 2011-01-17T06:39:00Z by zooko (previous) (diff)

comment:23 in reply to: ↑ 22 ; follow-up: Changed at 2011-01-17T07:34:13Z by davidsarah

Replying to zooko:

I agree that we shouldn't recommend easy_install allmydata-tahoe as a way to set up Tahoe-LAFS, for the reasons you gave.

Shouldn't recommend, or should recommend against?

I'd like us to recommend against.

In particular, this means that if there were another Python project which depended on Tahoe-LAFS, that project could mark Tahoe-LAFS as a requirement and then builds/installs of that project would automatically set up Tahoe-LAFS and all of its dependencies.

Yes, no problem with that. The bugs that are fixed in order to enable easy_install to work, are bugs that need fixing anyway.

comment:24 in reply to: ↑ 19 ; follow-up: Changed at 2011-01-17T13:49:50Z by bewst

Replying to davidsarah:

Be careful: if you use easy_install allmydata-tahoe to install into a directory where other Python packages that are Tahoe dependencies are installed, and then later try to run a newer version from a build using the newer version's bin/tahoe, then you may get an error because the older version is found by mistake. This shouldn't happen, but it can because of #1246.

(setup.py install has the same problem.)

You could always recommend that people always install in a virtualenv.

comment:25 in reply to: ↑ 24 Changed at 2011-01-18T00:51:18Z by davidsarah

Replying to bewst:

You could always recommend that people always install in a virtualenv.

There's no particular advantage to that (especially for people who aren't already using virtualenv) over always doing a build rather than an install.

comment:26 follow-up: Changed at 2011-01-18T01:18:09Z by bewst

I thought build was a subset of install. How do you get stuff into the PYTHONPATH, etc., if you just do a build? Manually?

comment:27 in reply to: ↑ 23 Changed at 2011-01-18T05:18:22Z by zooko

Replying to davidsarah:

Replying to zooko:

Shouldn't recommend, or should recommend against?

We should definitely focus of quickstart.html, and don't people alternatives to quickstart.html unless they specifically ask or have some specific needs that don't fit quickstart.html.

I personally wouldn't go so far as to recommend against using easy_install just because it is sort of beyond the scope of Tahoe-LAFS. Once we merge #1054 then Tahoe-LAFS works with easy_install at least as well as any other Python package does, and people have developed their own opinions and techniques about it. On the other hand, your specific reason to recommend against it is novel as far as I know and potentially quite useful knowledge to others. It isn't Tahoe-LAFS specific and might be best expressed in a bug report or an email message to distutils-sig.

So, I'm ±0 on dis-recommending easy_install allmydata-tahoe.

Version 0, edited at 2011-01-18T05:18:22Z by zooko (next)

comment:28 in reply to: ↑ 26 Changed at 2011-01-18T07:38:00Z by zooko

Replying to bewst:

I thought build was a subset of install. How do you get stuff into the PYTHONPATH, etc., if you just do a build? Manually?

When the user follows these instructions:

http://tahoe-lafs.org/source/tahoe-lafs/trunk/docs/quickstart.html

Then the "executable" that is created, named bin/tahoe (relative to the root of the tahoe-lafs source tree) sets up the necessary paths and so on.

I'm pretty pleased about this! I think it is nice that users can build and execute tahoe without having to "install" it, which usually requires greater privileges and can cause lots of collateral damage to their system. It is an example of the Principle of Least Authority. :-)

That doesn't mean, of course, that we try to make it hard to install if installing is what you really want to do. You can accomplish it in several ways, one of which is sudo python setup.py install.

comment:29 Changed at 2011-01-19T05:00:30Z by zooko

  • Keywords review-needed removed

comment:30 Changed at 2011-01-20T09:49:06Z by zooko

I will test this manually before the 1.8.2 final release.

comment:31 Changed at 2011-01-31T02:18:26Z by warner

  • Resolution set to fixed
  • Status changed from assigned to closed

we've manually tested this on ubuntu/maverick and ubuntu/lucid, and easy_install ./allmydata-tahoe-*.zip works. In addition, pip install ./tahoe.zip works on a maverick box. easy_install docs not work on a debian/lenny box because of some (Nevow-needs-zope.interface) problems that run into a bug in the older version of setuptools on lenny. Zooko tested easy_install on his OS-X box.

Good enough to ship!

Note: See TracTickets for help on using tickets.