easy_install allmydata-tahoe is broken

[francois]

Trying to install Tahoe 1.3.0 with easy_install fails with this message.

$ easy_install-2.5 --quiet --prefix $DIR allmydata-tahoe==1.3.0

The required version of setuptools (>=0.6c12dev) is not available, and
can't be installed while this script is running. Please install
 a more recent version first, using 'easy_install -U setuptools'.

(Currently using setuptools 0.6c8 (/usr/lib/python2.5/site-packages))
error: Setup script exited with 2

This was tested on Ubuntu intrepid and Debian lenny machines using Python 2.5.

However, installation from the tarball (python setup.py install) works fine.

[zooko]

That's because setuptools >= 0.6c12dev is actually zetuptoolz -- our own toothpick of setuptools: ​http://allmydata.org/trac/zetuptoolz

I hadn't realized that this breaks easy_install. I would like to fix this ASAP, but I'm not sure how.

[zooko]

Okay, I think I know how to fix this (just by renaming our version of setuptools to "zetuptoolz", but I don't understand why the current unit tests (such as this one: ​http://allmydata.org/buildbot/builders/Shawn%20intrepid%20amd64/builds/127/steps/install-egg/logs/stdio ) don't catch this bug, and I don't want to change the build/install code just before a release. So, tahoe 1.4.0 will ship with this bug and I'll move this ticket into the 1.4.1 Milestone.

[zooko]

This issue is causing at least the following failures to build/install:

​http://allmydata.org/buildbot/builders/zooko%20ootles%20Mac-amd64%2010.4/builds/196/steps/build/logs/stdio ​http://allmydata.org/buildbot-zfec/builders/BlackDew%20debian-unstable-i386/builds/10/steps/stdeb/logs/stdio

[ndurner]

From IRC:

(20:58:05) ndurner: Hmpf, my build instructions for Windows don't work anymore
(22:42:48) zooko: I think your report should go into ticket #668.

E:\allmydata-tahoe-1.4.0>python setup.py build
Not found: tahoe-deps
Not found: ../tahoe-deps
Traceback (most recent call last):
  File "setup.py", line 346, in <module>
    zip_safe=False, # We prefer unzipped for easier access.
  File "C:\Python25\lib\distutils\core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l
ine 219, in __init__
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l
ine 243, in fetch_build_eggs
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin
e 522, in resolve
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin
e 758, in best_match
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\pkg_resources.py", lin
e 770, in obtain
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\dist.py", l
ine 286, in fetch_build_egg
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 452, in easy_install
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 482, in install_item
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 661, in install_eggs
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 936, in build_and_install
  File "E:\allmydata-tahoe-1.4.0\setuptools-0.6c12dev.egg\setuptools\command\eas
y_install.py", line 927, in run_setup
distutils.errors.DistutilsError: Setup script exited with error: Python was buil
t with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

[zooko]

#721 was a duplicate of this (with useful information in the bug report).

[zooko]

Sigh. I hate to allow this issue to remain unfixed, but there really isn't a good reason to hold up the TahoeLAFS v1.5 release on it. The official install instructions ​http://allmydata.org/source/tahoe/trunk/docs/install.html currently work on all supported platforms as far as I know, and all of the "flunkOnFailure" building/packaging buildsteps on all of the SupportedBuilders? are passing.

Therefore, by invoking the definition of "Supported" to exclude everyone for whom this system does not currently work, I hereby bump this ticket out of the v1.5 Milestone. :-)

[bewst]

Replying to zooko:

The official install instructions ​http://allmydata.org/source/tahoe/trunk/docs/install.html currently work on all supported platforms as far as I know,

Unfortunately, it's failing for me on intrepid with an apparently-related symptom:

$ python setup.py build
running darcsver
setup.py darcsver: Failure from attempt to find version tags with 'darcs changes', and src/allmydata/_version.py already exists, so leaving it alone.
running develop
Checking .pth file support in support/lib/python2.5/site-packages
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 2] No such file or directory: 'support/lib/python2.5/site-packages/test-easy-install-8824.pth'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    support/lib/python2.5/site-packages

This directory does not currently exist.  Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).

Therefore, by invoking the definition of "Supported" to exclude everyone for whom this system does not currently work, I hereby bump this ticket out of the v1.5 Milestone. :-)

[zooko]

Thank you for your bug report. Your problem is related to but not the same as this one. I've opened a new ticket for your issue -- #803 (python ./setup.py install -- can't create or remove files in install directory).

[cgalvan]

[14:40] <RedG3> zooko: I switched to distribute and ran the test suite with tahoe and everything seems good to me :) I also did an easy_install of tahoe with distribute and it worked, but it installed setuptools since it's explicitly listed in the install_requires [14:41] <RedG3> I would like to test it on some other platforms, and if it works out I think we can safely change the install_requires to distribute if it is ok with you.

[zooko]

Replacing zetuptoolz with Distribute, as Chris is currently doing, might effect #709, #717, #756, #763, #799, and possibly quite a few others. :-)

[davidsarah]

setuptools delenda est.

[zooko]

My plan is to try replacing setuptools with distribute as well as submitting a couple of patches to distribute (​http://bitbucket.org/tarek/distribute/issue/136/respect-the-pythonpath and ​http://bitbucket.org/tarek/distribute/issue/46/be-more-like-distutils-with-regard-to ).

[zooko]

I'm very happy to report that since I updated pycryptopp to v0.5.28, that now sudo easy_install allmydata-tahoe works on my Mac OS X laptop even if I have none of the dependencies pre-installed [*].

Now if only we had an automated test which would go red if sudo easy_install allmydata-tahoe were to ever stop working...

(Probably it already worked before I updated pycryptopp if you were using easy_install as provided by setuptools, but if your easy_install is provided by distribute then you need ​the workaround in pycryptopp-0.5.28 for ​this distribute bug.)

([*] Except the pycrypto dependency doesn't work because ​http://pycrypto.org is unreachable (as it has been for days) but the PyPI record at ​http://pypi.python.org/pypi/pycrypto/2.3 shows ​http://www.pycrypto.org/files/pycrypto-2.3.tar.gz as the "Download URL" so easy_install always tries to download that file if it needs pycrypto and then aborts the Tahoe-LAFS build when it can't connect to pycrypto.org. The only work-around I can come up with for this at the moment is to add --find-links=http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-sdists/ to your easy_install command-line.)

[zooko]

Hooray! It works. I had to update pycryptopp and zfec to avoid build-dependencies, and someone (presumably dlitz) had to fix ​http://pycrypto.org to work again, but now easy_install allmydata-tahoe works! :-)

[zooko]

Now if only we had some sort of regression test of this functionality then I would be happy to close this ticket...

[davidsarah]

Replying to zooko:

Hooray! It works. I had to update pycryptopp and zfec to avoid build-dependencies, and someone (presumably dlitz) had to fix ​http://pycrypto.org to work again, but now easy_install allmydata-tahoe works! :-)

Be careful: if you use easy_install allmydata-tahoe to install into a directory where other Python packages that are Tahoe dependencies are installed, and then later try to run a newer version from a build using the newer version's bin/tahoe, then you may get an error because the older version is found by mistake. This shouldn't happen, but it can because of #1246 #1258.

(setup.py install has the same problem.)

I therefore think that we should be recommending against installing Tahoe to any site directories (which is where easy_install will put it by default). It should be safe to install it to a fresh directory using easy_install allmydata-tahoe --prefix=<some fresh directory>.

Also note that easy_install is some random version of setuptools or distribute, so who knows what bugs it has?

[zooko]

Oh, we also need to commit a fix to #1054 (ImportError?: No module named setuptools_darcs.setuptools_darcs) to trunk before this will work.

[zooko]

I agree that we shouldn't recommend easy_install allmydata-tahoe as a way to set up Tahoe-LAFS, for the reasons you gave. Also because there are other improvements that we've implemented in our primary, supported method of setting up--​quickstart.html--such as #1233 (prefer binary eggs which satisfy dependencies over source eggs which are even newer).

However, easy_install (and its cousin pip, which I hope also works now) are very popular, so I'm happy if Tahoe-LAFS is compatible with them.

In particular, this means that if there were another Python project which depended on Tahoe-LAFS, that project could mark Tahoe-LAFS as a requirement and then builds/installs of that project would automatically set up Tahoe-LAFS and all of its dependencies.

[davidsarah]

Replying to zooko:

I agree that we shouldn't recommend easy_install allmydata-tahoe as a way to set up Tahoe-LAFS, for the reasons you gave.

Shouldn't recommend, or should recommend against?

I'd like us to recommend against.

In particular, this means that if there were another Python project which depended on Tahoe-LAFS, that project could mark Tahoe-LAFS as a requirement and then builds/installs of that project would automatically set up Tahoe-LAFS and all of its dependencies.

Yes, no problem with that. The bugs that are fixed in order to enable easy_install to work, are bugs that need fixing anyway.

[bewst]

Replying to davidsarah:

Be careful: if you use easy_install allmydata-tahoe to install into a directory where other Python packages that are Tahoe dependencies are installed, and then later try to run a newer version from a build using the newer version's bin/tahoe, then you may get an error because the older version is found by mistake. This shouldn't happen, but it can because of #1246.

(setup.py install has the same problem.)

You could always recommend that people always install in a virtualenv.

[davidsarah]

Replying to bewst:

You could always recommend that people always install in a virtualenv.

There's no particular advantage to that (especially for people who aren't already using virtualenv) over always doing a build rather than an install.

[bewst]

I thought build was a subset of install. How do you get stuff into the PYTHONPATH, etc., if you just do a build? Manually?

[zooko]

Replying to davidsarah:

Replying to zooko:

Shouldn't recommend, or should recommend against?

We should definitely focus on ​quickstart.html, and don't offer people alternatives to quickstart.html unless they specifically ask or have some specific needs that don't fit quickstart.html.

I personally wouldn't go so far as to recommend against using easy_install just because it is sort of beyond the scope of Tahoe-LAFS. Once we merge #1054 then Tahoe-LAFS works with easy_install at least as well as any other Python package does, and people have developed their own opinions and techniques about it. On the other hand, ​your specific reason to recommend against it is novel as far as I know and potentially quite useful knowledge to others. It isn't Tahoe-LAFS specific and might be best expressed in a bug report or an email message to distutils-sig.

So, I'm ±0 on dis-recommending easy_install allmydata-tahoe.

[zooko]

Replying to bewst:

I thought build was a subset of install. How do you get stuff into the PYTHONPATH, etc., if you just do a build? Manually?

When the user follows these instructions:

​http://tahoe-lafs.org/source/tahoe-lafs/trunk/docs/quickstart.html

Then the "executable" that is created, named bin/tahoe (relative to the root of the tahoe-lafs source tree) sets up the necessary paths and so on.

I'm pretty pleased about this! I think it is nice that users can build and execute tahoe without having to "install" it, which usually requires greater privileges and can cause lots of collateral damage to their system. It is an example of the Principle of Least Authority. :-)

That doesn't mean, of course, that we try to make it hard to install if installing is what you really want to do. You can accomplish it in several ways, one of which is sudo python setup.py install.

[zooko]

I will test this manually before the 1.8.2 final release.

[warner]

we've manually tested this on ubuntu/maverick and ubuntu/lucid, and easy_install ./allmydata-tahoe-*.zip works. In addition, pip install ./tahoe.zip works on a maverick box. easy_install docs not work on a debian/lenny box because of some (Nevow-needs-zope.interface) problems that run into a bug in the older version of setuptools on lenny. Zooko tested easy_install on his OS-X box.

Good enough to ship!