make it work with bbfreeze

[zooko]

bbfreeze -- ​http://pypi.python.org/pypi/bbfreeze -- is an alternative to py2exe and (in the near future) to py2app. It is very easy to use:

easy_install bbfreeze
python ./setup.py bdist_bbfreeze

Except that it currently doesn't work for Tahoe on Windows -- something about our (very tricky and complicated) bin/tahoe hack is outwitting bbfreeze maybe?

[zooko]

Unassigning cgalvan since he hasn't been active in a couple of months.

[mids]

This works for me on 1.6.0:

python setup.py build
python setup.py install
cd windows
bb-freeze tahoe.py

tahoe.exe is created in dist directory and runs okay.

[davidsarah]

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

[davidsarah]

The exception when viewing the WUI was:

<type 'exceptions.WindowsError'>: [Error 2] The system cannot find the file specified:
'd:\\tahoe\\1.6.1\\tahoe-lafs\\windows\\dist\\allmydata_tahoe-1.6.0_r4256-py2.6.egg\\allmydata\\web\\directory.xhtml'

(That file indeed does not exist, although the directory does and contains .pyc files.)

zooko wrote on IRC:

[The .egg] is supposed to include [resource files] because (a) we said "include_package_data=True" as an argument to setup(), (b) that .xhtml file is registered under darcs revision control and (c) the setuptools_darcs plugin is installed.

[zooko]

Replying to davidsarah:

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

[davidsarah]

Replying to zooko:

Replying to davidsarah:

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

If it did before, it won't now, because the windows directory has been removed by #1282.

After much trial and error, I found the following recipe instead. First apply attachment:bb-freeze.darcs.patch​ to a checkout of trunk. Then from a cygwin prompt:

make clean
python setup.py build
sed 's/\\r//' <src/allmydata/_version.py >_version.new
mv -f _version.new src/allmydata/_version.py
export PYTHONPATH=support/Lib/site-packages
bb-freeze windows/tahoe.py

(cygwin is not essential, but we need sed to convert the CRLF line endings in _version.py, otherwise the compiler invoked by bb-freeze will fail to compile it. I don't know why it is so picky.)

The resulting dist\tahoe.exe has a bug that causes its argv array to be shifted, dropping sys.argv[0]. You can work around this by giving 'tahoe' as a dummy first argument.

I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI. Unicode arguments and output even work!

[davidsarah]

Various hacks to get bb-freeze to mostly work. Do not apply to trunk. refs #585

[davidsarah]

Various hacks to get bb-freeze to mostly work. Definitely do not apply to trunk. This version of the patch includes the workaround of adding as the dummy 'tahoe' argument so you don't need to. refs #585

[davidsarah]

Replying to davidsarah:

I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI.

And the WUI works without the resources problem in comment:5, I meant to say.

[davidsarah]

I worked out why the arguments are off-by-one -- it is because the code to retrieve Unicode arguments in src/allmydata/windows/fixups.py​ depends on the script having been invoked via the Python interpreter, so that unicode_argv[0] (the python executable) needs to be skipped. The executable produced by bbfreeze, OTOH, is not invoked via the Python interpreter. This problem wouldn't have occurred before the fix to #1074 in Tahoe v1.8.0, which explains why it worked for mids.

attachment:bb-freeze.darcs.3.patch​ (which depends on the patch for #1287) has now been updated to solve this problem in a way that doesn't interfere with the behaviour of the usual bin/tahoe.

This attachment also includes a patch to the bundled darcsver, to make it write _version.py with LF line endings, by changing "wt+" to "wb+" ​here. This matches all of the other Python source files, so should not cause any problems. (We should make a darcsver release, this patch to the bundled version is just for testing.)

windows/tahoe.py has been moved to static/tahoe.py because it is not specific to Windows. It should probably work for any ​static Python executable packager, although I haven't tested any others.

I've changed the patch to only omit the setuptools version and path from the output of get_package_versions_and_locations() when running the bb-frozen executable (which does not actually have setuptools as a runtime dependency). The problem that this is addressing is that if the executable tries to import setuptools, it can fail to find the original site module and exit at that point.

I also refactored a bunch of stuff in __init__.py and _auto_deps.py. get_package_versions_and_locations() is now in _auto_deps.py because it needs to be updated whenever install_requires is. It suppresses deprecation warnings while importing modules, so that the bb-frozen executable doesn't print these warnings. Finally, we now correctly check that the versions of imported modules actually satisfy the requirements (except for pyasn1, where we don't know the version number). Previously, require_auto_deps() only checked that the distributions that pkg_resources attempted to put on the sys.path satisfy the requirements that it was told to satisfy. That is not sufficient due to #1246, #1258 and possibly other bugs. I needed to simplify the pycrypto requirement to >= 2.3, since the new check doesn't support disjunctive requirements.

The recipe is now:

make clean
python setup.py build
export PYTHONPATH=support/Lib/site-packages
bb-freeze static/tahoe.py

(This should work on a Windows command prompt, modulo availability of 'make', and set instead of export.)

We need to be able to test the resulting executable. I will file another ticket for that.

[davidsarah]

Make bbfreeze work. This also refactors allmydata/init.py and _auto_deps.py and implements more robust checking of dependency requirements. It includes a change to the bundled darcsvar, to write the _version.py file with LF line endings.

[davidsarah]

static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

[davidsarah]

With attachment:bb-freeze.darcs.3.patch​ + attachment:static-tahoe.darcs.patch​ + attachment:ticket:1296:tahoe-debug-trial.darcs.patch​ , we pass everything except for skipping a few runner tests (which I will fix tomorrow).

[davidsarah]

Use dist\tahoe debug trial to test.

[david-sarah@…]

In [4938/ticket1306]:

Add static/tahoe.py, to support bbfreeze, py2exe etc. refs #585

[david-sarah@…]

In [4939/ticket1306]:

static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

[david-sarah@…]

In [4942/ticket1306]:

Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585

[david-sarah@…]

In [4943/ticket1306]:

Move get_package_versions_and_locations from init.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585

[david-sarah@…]

In 93c1909847e9dc44:

bin/tahoe-script.template, src/windows/fixups.py: simplify the method of stripping initial arguments in sys.argv on Windows. This helps with bb-freeze and running tahoe via 'coverage'. Also includes some wording changes and minor refactoring of bin/tahoe-script.template. refs #585, #1303

[zooko]

I reviewed 93c1909847e9dc44 and it looks good to me.

[zooko]

[20110113043040-93fa1-2a5ccfc48e2fe53b3fa5a91e913fbd300ccd98f4]/ticket1306 looks good. (I eyeballed it but didn't actually copy out the text and use diff to confirm that the text didn't get changed while it was getting moved.)

[zooko]

[20110107085259-93fa1-386c985e5bebe82119025222c9c39a43aa5b40b4]/ticket1306 looks fine.

[zooko]

Okay I read through [20110113042457-93fa1-b0c23a181f94c19e75407448bdd776c9a065f1a1]/ticket1306. I can't claim to understand it 100%, but I couldn't see any defect or think of any design problem with it. +0.9 :-)

[zooko]

David-Sarah is going to merge these patches into trunk.

[david-sarah@…]

In 8587c63bdc291faf:

Make bb-freeze (and probably other static packaging tools) work. This updates various places where we assumed that the tahoe process was executed via the Python interpreter. It also allows tests to recursively invoke the same tahoe.exe, rather than bin/tahoe. refs #585

[warner]

I'm guessing that 8587c63bdc291faf is meant to close this ticket. Please re-open if I'm wrong.

[warner]

davidsarah mentioned on IRC about an hour ago that there's one more patch to go in (".tac files import pkg_resources, so that's still a dependency"). Reopening for that.

[david-sarah@…]

In c8ccb297fa7f499d:

src/allmydata/_auto_deps.py: setuptools is still an install requirement even in frozen builds, because .tac files import pkg_resources. refs #585

[davidsarah]

The PYTHONPATH is not right in the recipe in comment:10. Using a Command Prompt, do:

python setup.py build
set PYTHONPATH=<tahoedir>\support\Lib\site-packages;<tahoedir>\src
bb-freeze static\tahoe.py

where <tahoedir> is the full path to the Tahoe source directory.

(make clean is only necessary if dependencies have changed.)

I'll add this to AdvancedInstall.