[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2729: Add binary builds to tahoe-lafs.org's buildbot/download page

Wed Feb 10 01:03:49 UTC 2016

#2729: Add binary builds to tahoe-lafs.org's buildbot/download page
-----------------------------+-----------------------------------
     Reporter:  cypher       |      Owner:  daira
         Type:  enhancement  |     Status:  new
     Priority:  normal       |  Milestone:  undecided
    Component:  packaging    |    Version:  1.10.2
   Resolution:               |   Keywords:  usability windows mac
Launchpad Bug:               |
-----------------------------+-----------------------------------
Description changed by cypher:

Old description:

> I've recently put together some simple build scripts to help automate the
> creation binary distributions of Tahoe-LAFS for Windows and Mac using
> [http://www.pyinstaller.org/ PyInstaller]. Like bb-freeze, py2exe,
> py2app, and other related projects, with enough tinkering, Pyinstaller
> can be used to create self-contained distributions of python programs
> that can effectively be run "out of the box" by end users. Given the
> rather convoluted state of python packaging (and with it, the additional
> mess of horrors that typically comes with building Tahoe-LAFS from
> scratch), I think it would be desirable to have such packages available
> to end users as an alternative to the current offering of a manual
> install.
>
> The aforementioned scripts can be found
> [https://github.com/gridsync/gridsync/blob/master/scripts/make-frozen-
> tahoe.bat here] (for Windows) and
> [https://github.com/gridsync/gridsync/blob/master/scripts/make-frozen-
> tahoe.sh here] (for OS X) and are currently being used by Gridsync's
> buildbot (which follows Tahoe-LAFS' upstream master branch and publishes
> the resultant binary packages [https://buildbot.gridsync.io/tahoe-lafs/
> here]). It shouldn't be terribly difficult to port these over to tahoe-
> lafs.org's buildbot and I'd be happy to help out with the process (or
> even lend some buildslaves to it, if needed).
>
> A few explanatory notes:
>
> * A ".spec" file is needed to inform Pyinstaller of any required files or
> modules that might not otherwise be detected by analyzing the package's
> import statements. The one I put together for tahoe
> [https://github.com/gridsync/gridsync/blob/master/tahoe.spec here] pulls
> in the static files needed for the web UI (as well as the
> 'characteristic' and 'pyasn1_modules' -- which, for whatever reason,
> aren't automatically picked up by Pyinstaller). If you use this, the
> specified filepaths will probably need to be changed.
>
> * The above spec file generates one-folder bundles by choice rather than
> one-file binaries since the one-file approach will fail if the system
> tempdir is mounted with the NOEXEC flag (and one-file mode works by
> extracting bundled files to the system tempdir and running them from
> there).
>
> * On Windows, OpenSSL will need to be installed in accordance with
> Daira's instructions outlined [https://github.com/tahoe-lafs/tahoe-
> lafs/blob/195.windows-packaging.10/docs/build/windows-installer.rst
> here]. I put together a quick script to do this
> [https://github.com/gridsync/gridsync/blob/master/scripts/make-tahoe-
> deps.bat here] but the dependencies specified at the top of the file will
> still need to be installed manually.
>
> * The build scripts contain a few workarounds that seem to be necessary
> to pass the dependency checks when running frozen. This includes
> [https://github.com/gridsync/gridsync/blob/master/disable_setuptools.patch
> patching out the setuptools requirement from _auto_deps.py] and juggling
> versions of Twisted on Windows to appease nevow (whose install fails for
> me when using the pinned version of Twisted). Maybe this isn't necessary
> and maybe there's a better way to do it but I couldn't get tahoe running
> frozen without it and, frankly, there's a lot going on with Tahoe-LAFS'
> auto_deps/setuptools that I don't quite understand (and I look forward to
> the day when ticket #1582 is closed). :)
>
> * Pyinstaller builds can be made reproducible (assuming the same
> platform, interpreter version, and architecture) by setting the
> PYTHONHASHSEED environment variable to a known/shared value. Note that
> this only affects the pythonic bits (and not, e.g., the C libraries);
> given the current dependencies, more work is needed to make these
> packages fully reproducible (but even so remains a worthy goal).
>
> * Pyinstaller builds are forwards-compatible but not (always) backwards-
> compatible; it's better to build on older operating system versions if
> possible.

New description:

 I've recently put together some simple build scripts to help automate the
 creation binary distributions of Tahoe-LAFS for Windows and Mac using
 [http://www.pyinstaller.org/ PyInstaller]. Like bb-freeze, py2exe, py2app,
 and other related projects, with enough tinkering, Pyinstaller can be used
 to create self-contained distributions of python programs that can
 effectively be run "out of the box" by end users. Given the rather
 convoluted state of python packaging (and with it, the additional mess of
 horrors that typically comes with building Tahoe-LAFS from scratch), I
 think it would be desirable to have such packages available to end users
 as an alternative to the current offering of a manual install.

 The aforementioned scripts can be found
 [https://github.com/gridsync/gridsync/blob/master/scripts/make-frozen-
 tahoe.bat here] (for Windows) and
 [https://github.com/gridsync/gridsync/blob/master/scripts/make-frozen-
 tahoe.sh here] (for OS X) and are currently being used by Gridsync's
 buildbot (which follows Tahoe-LAFS' upstream master branch and publishes
 the resultant binary packages [https://buildbot.gridsync.io/tahoe-lafs/
 here]). It shouldn't be terribly difficult to port these over to tahoe-
 lafs.org's buildbot and I'd be happy to help out with the process (or even
 lend some buildslaves to it, if needed).

 A few explanatory notes:

 * A ".spec" file is needed to inform Pyinstaller of any required files or
 modules that might not otherwise be detected by analyzing the package's
 import statements. The one I put together for tahoe
 [https://github.com/gridsync/gridsync/blob/master/tahoe.spec here] pulls
 in the static files needed for the web UI (as well as the 'characteristic'
 and 'pyasn1_modules' -- which, for whatever reason, aren't automatically
 picked up by Pyinstaller). If you use this, the specified filepaths will
 probably need to be changed.

 * The above spec file generates one-folder bundles by choice rather than
 one-file binaries since the one-file approach will fail if the system
 tempdir is mounted with the NOEXEC flag (and one-file mode works by
 extracting bundled files to the system tempdir and running them from
 there).

 * On Windows, OpenSSL will need to be installed in accordance with Daira's
 instructions outlined [https://github.com/tahoe-lafs/tahoe-lafs/blob/195
 .windows-packaging.10/docs/build/windows-installer.rst here]. I put
 together a quick script to do this
 [https://github.com/gridsync/gridsync/blob/master/scripts/make-tahoe-
 deps.bat here] but the dependencies specified at the top of the file will
 still need to be installed manually.

 * The build scripts
 [https://github.com/gridsync/gridsync/blob/master/disable_setuptools.patch
 patch out the setuptools requirement from _auto_deps.py] before running
 pyinstaller which allows tahoe to run frozen. Maybe this isn't necessary
 and maybe there's a better way to do it (I do see [https://github.com
 /tahoe-lafs/tahoe-lafs/blob/master/src/allmydata/_auto_deps.py#L100 some
 code to this effect in _auto_deps.py]) but I couldn't get tahoe running
 frozen without patching as __init.py__'s check_requirement() fails with
 "PackagingError: no version info for setuptools". More investigation is
 needed..

 * Pyinstaller builds can be made reproducible (assuming the same platform,
 interpreter version, and architecture) by setting the PYTHONHASHSEED
 environment variable to a known/shared value. Note that this only affects
 the pythonic bits (and not, e.g., the C libraries); given the current
 dependencies, more work is needed to make these packages fully
 reproducible (but even so remains a worthy goal).

 * Pyinstaller builds are forwards-compatible but not (always) backwards-
 compatible; it's better to build on older operating system versions if
 possible.

--

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2729#comment:2>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage