#585 closed enhancement (fixed)

make it work with bbfreeze

Reported by: zooko Owned by: davidsarah
Priority: major Milestone: 1.8.2
Component: packaging Version: 1.2.0
Keywords: windows install setuptools bbfreeze reviewed Cc: slush@…, slush
Launchpad Bug:

Description

bbfreeze -- http://pypi.python.org/pypi/bbfreeze -- is an alternative to py2exe and (in the near future) to py2app. It is very easy to use:

easy_install bbfreeze
python ./setup.py bdist_bbfreeze

Except that it currently doesn't work for Tahoe on Windows -- something about our (very tricky and complicated) bin/tahoe hack is outwitting bbfreeze maybe?

Attachments (4)

bb-freeze.darcs.patch (13.1 KB) - added by davidsarah at 2011-01-03T07:01:50Z.
Various hacks to get bb-freeze to mostly work. Do not apply to trunk. refs #585
bb-freeze.darcs.2.patch (13.6 KB) - added by davidsarah at 2011-01-03T07:14:24Z.
Various hacks to get bb-freeze to mostly work. Definitely do not apply to trunk. This version of the patch includes the workaround of adding as the dummy 'tahoe' argument so you don't need to. refs #585
bb-freeze.darcs.3.patch (40.4 KB) - added by davidsarah at 2011-01-04T12:05:58Z.
Make bbfreeze work. This also refactors allmydata/init.py and _auto_deps.py and implements more robust checking of dependency requirements. It includes a change to the bundled darcsvar, to write the _version.py file with LF line endings.
static-tahoe.darcs.patch (11.8 KB) - added by davidsarah at 2011-01-07T10:41:19Z.
static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

Download all attachments as: .zip

Change History (35)

comment:1 Changed at 2009-04-07T03:02:38Z by zooko

  • Owner changed from cgalvan to nobody

Unassigning cgalvan since he hasn't been active in a couple of months.

comment:2 Changed at 2009-11-08T01:08:12Z by davidsarah

  • Keywords windows added
  • Type changed from defect to enhancement

comment:3 Changed at 2009-12-12T02:52:06Z by davidsarah

  • Keywords install added

comment:4 Changed at 2010-02-27T19:29:37Z by mids

This works for me on 1.6.0:

python setup.py build
python setup.py install
cd windows
bb-freeze tahoe.py

tahoe.exe is created in dist directory and runs okay.

comment:5 follow-up: Changed at 2010-03-02T00:57:05Z by davidsarah

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

comment:6 Changed at 2010-03-02T01:01:29Z by davidsarah

The exception when viewing the WUI was:

<type 'exceptions.WindowsError'>: [Error 2] The system cannot find the file specified:
'd:\\tahoe\\1.6.1\\tahoe-lafs\\windows\\dist\\allmydata_tahoe-1.6.0_r4256-py2.6.egg\\allmydata\\web\\directory.xhtml'

(That file indeed does not exist, although the directory does and contains .pyc files.)

zooko wrote on IRC:

[The .egg] is supposed to include [resource files] because (a) we said "include_package_data=True" as an argument to setup(), (b) that .xhtml file is registered under darcs revision control and (c) the setuptools_darcs plugin is installed.

comment:7 in reply to: ↑ 5 ; follow-up: Changed at 2010-06-21T21:02:37Z by zooko

  • Cc slush@… slush added
  • Keywords setuptools bbfreeze added
  • Milestone changed from undecided to soon

Replying to davidsarah:

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

comment:8 in reply to: ↑ 7 ; follow-up: Changed at 2011-01-03T07:01:03Z by davidsarah

Replying to zooko:

Replying to davidsarah:

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

If it did before, it won't now, because the windows directory has been removed by #1282.

After much trial and error, I found the following recipe instead. First apply attachment:bb-freeze.darcs.patch to a checkout of trunk. Then from a cygwin prompt:

make clean
python setup.py build
sed 's/\\r//' <src/allmydata/_version.py >_version.new
mv -f _version.new src/allmydata/_version.py
export PYTHONPATH=support/Lib/site-packages
bb-freeze windows/tahoe.py

(cygwin is not essential, but we need sed to convert the CRLF line endings in _version.py, otherwise the compiler invoked by bb-freeze will fail to compile it. I don't know why it is so picky.)

The resulting dist\tahoe.exe has a bug that causes its argv array to be shifted, dropping sys.argv[0]. You can work around this by giving 'tahoe' as a dummy first argument.

I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI. Unicode arguments and output even work!

Changed at 2011-01-03T07:01:50Z by davidsarah

Various hacks to get bb-freeze to mostly work. Do not apply to trunk. refs #585

Changed at 2011-01-03T07:14:24Z by davidsarah

Various hacks to get bb-freeze to mostly work. Definitely do not apply to trunk. This version of the patch includes the workaround of adding as the dummy 'tahoe' argument so you don't need to. refs #585

comment:9 in reply to: ↑ 8 Changed at 2011-01-03T07:20:45Z by davidsarah

Replying to davidsarah:

I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI.

And the WUI works without the resources problem in comment:5, I meant to say.

comment:10 Changed at 2011-01-04T12:03:24Z by davidsarah

I worked out why the arguments are off-by-one -- it is because the code to retrieve Unicode arguments in src/allmydata/windows/fixups.py depends on the script having been invoked via the Python interpreter, so that unicode_argv[0] (the python executable) needs to be skipped. The executable produced by bbfreeze, OTOH, is not invoked via the Python interpreter. This problem wouldn't have occurred before the fix to #1074 in Tahoe v1.8.0, which explains why it worked for mids.

attachment:bb-freeze.darcs.3.patch (which depends on the patch for #1287) has now been updated to solve this problem in a way that doesn't interfere with the behaviour of the usual bin/tahoe.

This attachment also includes a patch to the bundled darcsver, to make it write _version.py with LF line endings, by changing "wt+" to "wb+" here. This matches all of the other Python source files, so should not cause any problems. (We should make a darcsver release, this patch to the bundled version is just for testing.)

windows/tahoe.py has been moved to static/tahoe.py because it is not specific to Windows. It should probably work for any static Python executable packager, although I haven't tested any others.

I've changed the patch to only omit the setuptools version and path from the output of get_package_versions_and_locations() when running the bb-frozen executable (which does not actually have setuptools as a runtime dependency). The problem that this is addressing is that if the executable tries to import setuptools, it can fail to find the original site module and exit at that point.

I also refactored a bunch of stuff in __init__.py and _auto_deps.py. get_package_versions_and_locations() is now in _auto_deps.py because it needs to be updated whenever install_requires is. It suppresses deprecation warnings while importing modules, so that the bb-frozen executable doesn't print these warnings. Finally, we now correctly check that the versions of imported modules actually satisfy the requirements (except for pyasn1, where we don't know the version number). Previously, require_auto_deps() only checked that the distributions that pkg_resources attempted to put on the sys.path satisfy the requirements that it was told to satisfy. That is not sufficient due to #1246, #1258 and possibly other bugs. I needed to simplify the pycrypto requirement to >= 2.3, since the new check doesn't support disjunctive requirements.

The recipe is now:

make clean
python setup.py build
export PYTHONPATH=support/Lib/site-packages
bb-freeze static/tahoe.py

(This should work on a Windows command prompt, modulo availability of 'make', and set instead of export.)

We need to be able to test the resulting executable. I will file another ticket for that.

Changed at 2011-01-04T12:05:58Z by davidsarah

Make bbfreeze work. This also refactors allmydata/init.py and _auto_deps.py and implements more robust checking of dependency requirements. It includes a change to the bundled darcsvar, to write the _version.py file with LF line endings.

comment:11 Changed at 2011-01-04T12:08:58Z by davidsarah

  • Keywords review-needed added
  • Milestone changed from soon to 1.9.0

Changed at 2011-01-07T10:41:19Z by davidsarah

static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

comment:12 Changed at 2011-01-07T10:44:34Z by davidsarah

With attachment:bb-freeze.darcs.3.patch + attachment:static-tahoe.darcs.patch + attachment:ticket:1296:tahoe-debug-trial.darcs.patch , we pass everything except for skipping a few runner tests (which I will fix tomorrow).

comment:13 Changed at 2011-01-07T10:46:30Z by davidsarah

Use dist\tahoe debug trial to test.

Last edited at 2011-01-07T10:46:47Z by davidsarah (previous) (diff)

comment:14 Changed at 2011-01-14T23:29:23Z by zooko

  • Owner changed from nobody to zooko
  • Status changed from new to assigned

comment:15 Changed at 2011-01-15T05:14:15Z by david-sarah@…

In [4938/ticket1306]:

Add static/tahoe.py, to support bbfreeze, py2exe etc. refs #585

comment:16 Changed at 2011-01-15T05:14:16Z by david-sarah@…

In [4939/ticket1306]:

static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

comment:17 Changed at 2011-01-15T05:14:17Z by david-sarah@…

In [4942/ticket1306]:

Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585

comment:18 Changed at 2011-01-15T05:14:17Z by david-sarah@…

In [4943/ticket1306]:

Move get_package_versions_and_locations from init.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585

comment:19 Changed at 2011-01-16T21:25:15Z by davidsarah

  • Milestone changed from 1.9.0 to 1.8.2

comment:20 Changed at 2011-01-19T05:40:51Z by david-sarah@…

In 93c1909847e9dc44:

bin/tahoe-script.template, src/windows/fixups.py: simplify the method of stripping initial arguments in sys.argv on Windows. This helps with bb-freeze and running tahoe via 'coverage'. Also includes some wording changes and minor refactoring of bin/tahoe-script.template. refs #585, #1303

comment:21 Changed at 2011-01-19T06:12:41Z by zooko

I reviewed 93c1909847e9dc44 and it looks good to me.

comment:22 Changed at 2011-01-19T07:39:26Z by zooko

[20110113043040-93fa1-2a5ccfc48e2fe53b3fa5a91e913fbd300ccd98f4]/trunk looks good. (I eyeballed it but didn't actually copy out the text and use diff to confirm that the text didn't get changed while it was getting moved.)

Version 0, edited at 2011-01-19T07:39:26Z by zooko (next)

comment:24 Changed at 2011-01-19T07:48:40Z by zooko

Okay I read through [20110113042457-93fa1-b0c23a181f94c19e75407448bdd776c9a065f1a1]/ticket1306. I can't claim to understand it 100%, but I couldn't see any defect or think of any design problem with it. +0.9 :-)

comment:25 Changed at 2011-01-19T07:51:03Z by zooko

  • Keywords reviewed added; review-needed removed
  • Owner changed from zooko to warner
  • Status changed from assigned to new

comment:26 Changed at 2011-01-20T09:37:36Z by zooko

  • Owner changed from warner to davidsarah

David-Sarah is going to merge these patches into trunk.

comment:27 Changed at 2011-01-21T09:00:03Z by david-sarah@…

In 8587c63bdc291faf:

Make bb-freeze (and probably other static packaging tools) work. This updates various places where we assumed that the tahoe process was executed via the Python interpreter. It also allows tests to recursively invoke the same tahoe.exe, rather than bin/tahoe. refs #585

comment:28 Changed at 2011-01-21T17:22:45Z by warner

  • Resolution set to fixed
  • Status changed from new to closed

I'm guessing that 8587c63bdc291faf is meant to close this ticket. Please re-open if I'm wrong.

comment:29 Changed at 2011-01-21T23:54:09Z by warner

  • Resolution fixed deleted
  • Status changed from closed to reopened

davidsarah mentioned on IRC about an hour ago that there's one more patch to go in (".tac files import pkg_resources, so that's still a dependency"). Reopening for that.

comment:30 Changed at 2011-01-22T03:42:30Z by david-sarah@…

In c8ccb297fa7f499d:

src/allmydata/_auto_deps.py: setuptools is still an install requirement even in frozen builds, because .tac files import pkg_resources. refs #585

comment:31 Changed at 2011-01-22T04:57:17Z by davidsarah

  • Resolution set to fixed
  • Status changed from reopened to closed

The PYTHONPATH is not right in the recipe in comment:10. Using a Command Prompt, do:

python setup.py build
set PYTHONPATH=<tahoedir>\support\Lib\site-packages;<tahoedir>\src
bb-freeze static\tahoe.py

where <tahoedir> is the full path to the Tahoe source directory.

(make clean is only necessary if dependencies have changed.)

I'll add this to AdvancedInstall.

Note: See TracTickets for help on using tickets.