[tahoe-lafs-trac-stream] [Tahoe-LAFS] #1260: there should be a way to force compiling C extensions from source

[Tahoe-LAFS]

#1260: there should be a way to force compiling C extensions from source
----------------------------+------------------------------------------
     Reporter:  davidsarah  |      Owner:  somebody
         Type:  defect      |     Status:  closed
     Priority:  major       |  Milestone:  1.11.0
    Component:  packaging   |    Version:  1.8.0
   Resolution:  fixed       |   Keywords:  install setuptools docs test
Launchpad Bug:              |
----------------------------+------------------------------------------
Changes (by warner):

 * status:  new => closed
 * resolution:   => fixed
 * milestone:  undecided => 1.11.0


Old description:

> On tahoe-dev, Zooko O'Whielacronx wrote:
> > One of the goals for this release is to let people on more platforms
> more easily set up Tahoe-LAFS. (Problems with the setup process are
> probably the number one complaint mentioned on the mailing list and issue
> tracker right now.)
>
> > Tahoe-LAFS itself is pure Python and doesn't need to be compiled at
> all, but some of its dependencies do, so a big part of the "ease of
> setup" goal is to provide binary packages of those dependencies.
>
> Scott Dial replied:
> > I actually found this release '''less''' easy to install. On my linux
> machines, neither the {{{setup.py}}} or using {{{easy_install}}} yielded
> working dependencies for my system. Eventually I tracked it down to the
> {{{pycryptopp}}} binary egg:
>
> [snip problem due to a {{{libstdc++}}} ABI mismatch]
>
> > After solving the mystery, I was forced to resort to introspection:
> {{{
> $ python -c "import sys; \
>            sys.path.insert(0, 'src/allmydata'); \
>            from _auto_deps import install_requires; \
>             print '\n'.join(install_requires)" \
>   | xargs -I{} pip install "{}"
> # This dependency gets missed for some reason:
> $ pip install pyOpenSSL
> }}}
>
> > To get all of the dependencies to be built by my system
> ({{{easy_install}}} will use the binary eggs, and there is no way to
> force it to ignore them), {{{pip}}} was the only tool I could find that
> would do it without me being involved.
>
> > This release took infinitely longer to install than 1.7, where I just
> ran "{{{python setup.py build}}}" and "{{{python setup.py install}}}" and
> was done.
>
> We need to support an option to force compiling the C/C++ code in
> dependencies from source, for users who either encounter problems with
> the binary eggs, or just prefer to do it that way.

New description:

 On tahoe-dev, Zooko O'Whielacronx wrote:
 > One of the goals for this release is to let people on more platforms
 more easily set up Tahoe-LAFS. (Problems with the setup process are
 probably the number one complaint mentioned on the mailing list and issue
 tracker right now.)

 > Tahoe-LAFS itself is pure Python and doesn't need to be compiled at all,
 but some of its dependencies do, so a big part of the "ease of setup" goal
 is to provide binary packages of those dependencies.

 Scott Dial replied:
 > I actually found this release '''less''' easy to install. On my linux
 machines, neither the {{{setup.py}}} or using {{{easy_install}}} yielded
 working dependencies for my system. Eventually I tracked it down to the
 {{{pycryptopp}}} binary egg:

 [snip problem due to a {{{libstdc++}}} ABI mismatch]

 > After solving the mystery, I was forced to resort to introspection:
 {{{
 $ python -c "import sys; \
            sys.path.insert(0, 'src/allmydata'); \
            from _auto_deps import install_requires; \
             print '\n'.join(install_requires)" \
   | xargs -I{} pip install "{}"
 # This dependency gets missed for some reason:
 $ pip install pyOpenSSL
 }}}

 > To get all of the dependencies to be built by my system
 ({{{easy_install}}} will use the binary eggs, and there is no way to force
 it to ignore them), {{{pip}}} was the only tool I could find that would do
 it without me being involved.

 > This release took infinitely longer to install than 1.7, where I just
 ran "{{{python setup.py build}}}" and "{{{python setup.py install}}}" and
 was done.

 We need to support an option to force compiling the C/C++ code in
 dependencies from source, for users who either encounter problems with the
 binary eggs, or just prefer to do it that way.

--

Comment:

 We're using wheels now, but to force pip to compile from source even
 though a binary wheel is available, use: e.g. `pip wheel --no-use-wheel
 zfec`.

 Also note that neither we nor PyPI host *linux* binary wheels, because of
 the variety of incompatible ABIs in the linux world. There is work (see
 the [https://www.python.org/dev/peps/pep-0513/ PEP-513 "manylinux"] spec)
 to fix this, but we're not using it yet.

 I think we can close this. Please re-open if you disagree.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1260#comment:1>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage