users expect 'python setup.py install' to install dependencies

[sel]

Problem after installation in Fedora 19:

$ tahoe(most recent call last):
  File "/usr/bin/tahoe", line 6, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

[daira]

What command was used to install Tahoe?

[sel]

su -c 'python setup.py install'

[sel]

a list of all dependencies should do, we can then install them with a single command.

[gdt]

I think part of the point of the bug is that user expectations are not necessarily right. There are basically three approaches

• use a package from a packaging system. Users should be pointed to this as the standard approach. The overwhelming majority of uses of successful programs are like this.
• run some command from sources and have it go find and install dependencies (presuambly only some, and not thing like python!!)
• read the README, install the dependencies from a packaging system, and then build/install. This is the standard approach for people who are going to do development rather than just run.

My only concern is that the auto-install-dependencies approach is actively harmful for packaging systems, because package builds should not use the net and should be 100% byte-for-byte repeatable. So getting dependencies should be an explicit step the user has to ask for (perhaps this can be hinted at in the error if they aren't there).

[sel]

The separate step approach is unusual in fedora/centos, unless you are building from sources, in which case you should be prepared to build/install every required package. The situation I found: 1) Some dependencies were discovered during the 'build' process, 2) Some other dependencies were discovered after the 'install' process, when actually running the tahoe binary.

[gdt]

@sel: I am having trouble following you. Are you talking about somehow who is building a tahoe-lafs package using packaging control scripts? Or a user not using the packaging system who has downloaded source. Surely it's normal to have to have the prereqs installed first; you can't build emacs without first having gtk2+ (well, assuming you want it built that way, as an example).

In pkgsrc, this is handled by declaring all the things tahoe needs as dependencies, so those packages are installed before the build starts. Doesn't RHEL/FC/CentOS do the same thing?

[sel]

I downloaded the .zip unpacked it and followed the instructions. I didn't find a list of dependencies to install, I have discovered them from error messages by running the build multiple times, and after the install by running tahoe multiple times.

One thing: right now fedora has mock-1.1.32, but tahoe did not run until I installed mock-0.8.0. This is a downgrade with respect to the packaging system.

[gdt]

So perhaps you should open a new ticket "The build instructions do not clearly specify the list of dependencies", which should be easy to fix. I admit to not having read them in a really long time.

[sel]

Replying to gdt:

@sel: I am having trouble following you. Are you talking about somehow who is building a tahoe-lafs package using packaging control scripts? Or a user not using the packaging system who has downloaded source. Surely it's normal to have to have the prereqs installed first; you can't build emacs without first having gtk2+ (well, assuming you want it built that way, as an example).

In pkgsrc, this is handled by declaring all the things tahoe needs as dependencies, so those packages are installed before the build starts. Doesn't RHEL/FC/CentOS do the same thing?

It should do the same thing, however the packaging is different from Debian. RHEL/FC/CentOS uses yum, which wants a .rpm

[sel]

Replying to gdt:

I think part of the point of the bug is that user expectations are not necessarily right. There are basically three approaches

• use a package from a packaging system. Users should be pointed to this as the standard approach. The overwhelming majority of uses of successful programs are like this.
• run some command from sources and have it go find and install dependencies (presuambly only some, and not thing like python!!)
• read the README, install the dependencies from a packaging system, and then build/install. This is the standard approach for people who are going to do development rather than just run.

My only concern is that the auto-install-dependencies approach is actively harmful for packaging systems, because package builds should not use the net and should be 100% byte-for-byte repeatable. So getting dependencies should be an explicit step the user has to ask for (perhaps this can be hinted at in the error if they aren't there).

Generally I think the package system just downloads *prebuilt* packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.
My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible.

[zooko]

I agree that it is a problem that users expect python setup.py install to install dependencies, but it doesn't do that. The fact that it doesn't do that is tracked by ticket #1494. The fact that users decide to try python setup.py install is the subject of this ticket. (I think some of gdt's comments from comment:5 are the subject of other tickets, and it would probably help if gdt would find those tickets and cross-link them to this one.)

So: one possible patch to help with this ticket would be to add a line to quickstart.rst saying something like:

Note: running python setup.py install will copy only the Tahoe-LAFS package and not any of the dependent packages into your system. We don't currently support any way to install Tahoe-LAFS into your system. Instead, we recommend that you run it from the directory you built it in, as described in this document. See also the instructions for getting prebuilt packages from your operating system: ​https://tahoe-lafs.org/trac/tahoe-lafs/wiki/Installation

What do you think? Would text like that make it so people who read quickstart.rst stop trying python setup.py install?

(Then, if anyone ever fixes #1494 and makes python setup.py install work, we could remove this text.)

[daira]

See also #1494.

[sel]

just one quick comment: The package names in Debian repositories are different from the package names in other distributions such as Fedora/CentOS. Generally Fedora package names have the 'python-' prefix.

[gdt]

Replying to sel:

Generally I think the package system just downloads *prebuilt* packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.
My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible.

Sorry, I wasn't clear enough. There are two parts to packaging systems. One is the control files that take the upstream distribution file and build a binary package. The other is how users install binary packages. Both of these mechanisms are within the packaging system, and both have to manage dependencies by expressing that other packages are required first.

It seems that in most GNU/Linux systems, users almost never build packages from source using the packaging system, but just use binary packages. In pkgsrc, the norm is to build from source but using the packaging system, because of the huge number of OS/os-version/CPU tuples.

I think it's an artifact of tahoe being in python that there is this focus on setup scripts automatically fetching and installing dependencies. In general, step one of building something from source is to read the dependency list and make sure those are installed. So I think the focus should be on having tahoe available via packaging systems, rather than out-of-system dependency management. (I've done this for pkgsrc, and thus am mostly ignoring the setuptools kerfluffle.)

[sel]

Replying to gdt:

Replying to sel:

Generally I think the package system just downloads *prebuilt* packages. It is okay to have a command line option that must be specified to install dependencies, it is also okay to have a list of dependencies that can be installed in a separate step.
My concern is: once I have created the introducer, I need to create N storage nodes, that's why I would like to automate the installation as much as possible.

Sorry, I wasn't clear enough. There are two parts to packaging systems. One is the control files that take the upstream distribution file and build a binary package. The other is how users install binary packages. Both of these mechanisms are within the packaging system, and both have to manage dependencies by expressing that other packages are required first.

It seems that in most GNU/Linux systems, users almost never build packages from source using the packaging system, but just use binary packages. In pkgsrc, the norm is to build from source but using the packaging system, because of the huge number of OS/os-version/CPU tuples.

I think that pkgsrc and yum are different, afaik yum uses separate packages for binaries (rpm) or sources (srpm), but I have never seen yum build packages from source.

I think it's an artifact of tahoe being in python that there is this focus on setup scripts automatically fetching and installing dependencies. In general, step one of building something from source is to read the dependency list and make sure those are installed. So I think the focus should be on having tahoe available via packaging systems, rather than out-of-system dependency management. (I've done this for pkgsrc, and thus am mostly ignoring the setuptools kerfluffle.)

There are a couple of situations where the packaging system doesn't help: for example the mock package (needed by tahoe) that comes with current fedora, is higher version (1.1.32) from the one needed by tahoe (0.8.0).

[zooko]

Folks, instead of fixing it so that users don't expect automatic dependency-resolution from python setup.py install, we're going to fix it so that users get what they expect. That is the subject of ticket #1494. I'm closing this ticket as wontfix. If you want to push forward some other issues, for example things like "disallow automatic download of dependencies", please find the ticket about that and add a comment on that ticket.

[daira]

Replying to zooko:

Folks, instead of fixing it so that users don't expect automatic dependency-resolution from python setup.py install, we're going to fix it so that users get what they expect.

+1. The only reason this ticket was filed was that I was under the impression that not installing dependencies was standard behaviour for setup.py install, but apparently it's a regression in zetuptoolz.

That is the subject of ticket #1494.

Indeed.

If you want to push forward some other issues, for example things like "disallow automatic download of dependencies", please find the ticket about that and add a comment on that ticket.

That's ticket #1220.