build a .pkg installer for Mac OS X 10.9 Mavericks (intel-x86-64)

[zooko]

My Mom wants to see some home movies of my children playing in the back yard. Even though our docs/quickstart.rst is very well written, it is still far beyond her ken to build Tahoe from source. How could she get Tahoe installed on her home computer, which is a PowerPC G5 running Mac OS 10.3.9?

One way is that I could upload a binary .egg and instruct her in how to install Python 2.5.1 and easy_install and then how to run "easy_install allmydata-tahoe". However, the dependent libraries like Nevow and Twisted and so forth wouldn't get automatically installed by easy_install so that wouldn't quite work.

Another way is that I could build binary .egg packages of all of our dependent libraries and host them somewhere or bundle them into the tahoe binary .egg.

Another way is that we could use the "py2app" feature to build an installer than works on Mac OS 10.

[zooko]

The py2app approach still requires that we bundle binaries of all of our dependent libraries.

[zooko]

So perhaps a next-step on this is to experiment with making a complete bundle of Tahoe and all dependencies. :-) I did that for Zfone and then used py2app to build a Mac .app of the result.

[warner]

The py2app approach would require those dependent libraries to be present on the package-builder's machine, but the resulting package would be completely standalone, right?

I've started playing with py2app and have run into some problems (it seems to refuse to include foolscap in the bundle, despite being explicitly imported by the main entry-point script, even though it includes zfec and twisted just fine). But overall I think it will work.

So my main question is: what exactly should this packaged application look like?

One possibility:

• the tahoe node is created by the install process (using one of those fancy Assistants)
• the node is *not* started at boot time
• when the user double-clicks on the tahoe icon, their node is started, and a small window is created
  • this window has a button which, when pressed, sends their web browser to the local start page
  • dropping a URI in the window sends the web browser to the corresponding /URI/NNN page
  • dropping a file in the window causes the file to be uploaded, and the URI is displayed in a way that allows it to be cut-and-pasted into another application

thoughts?

[zooko]

The py2app approach would require those dependent libraries to be present on the package-builder's machine, but the resulting package would be completely standalone, right?

Right.

the tahoe node is created by the install process (using one of those fancy Assistants)

A Mac OS X idiom which I love is that installing is merely dragging and dropping (i.e. copying or moving) the Thingie.app into your filesystem. This pattern abjures running an "installer" (I'm not sure if that's related to an "Assistant") at install-time. I hope we use this pattern!

That would imply that node-creation happens at runtime instead, for example "if there is not already a node in ~/.tahoe".

[terrell]

additionally, there should be a prominent place for picking which network to be a part of (friendnet, testgrid, allmydata public grid, etc). Is that a dropdown with the easy choices? A means to discover their friend's introducer.furl?

That would imply that node-creation happens at runtime instead, for example "if there is not already a node in ~/.tahoe".

yes, totally.

later, create some kind of manager/preferences for multiple nodes, but initially, just create the one if it doesn't exist. if it does already exist, have a suppressible popup that would allow an overwrite/cleanstart.

[geal]

Hi,

I can help a bit on this bug. I have already messed with py2app to generate .app packages and I have some code to automatically generate DMG images.

I first tried to add some basic options to py2app and ran it, to see if everything is ok. Unfortunately, it seems the latest version of modulegraph (necessary for py2app) is in conflict with the setuptools version that is used in the project (ImpLoader is part of pkg_resources).

running py2app
running build_py
copying src/allmydata/_version.py -> build/bdist.macosx-10.6-intel/lib/allmydata
creating /Users/geoffroycouprie/dev/python/tahoe-lafs/build/bdist.macosx-10.6-intel/python2.7-standalone/app
creating /Users/geoffroycouprie/dev/python/tahoe-lafs/build/bdist.macosx-10.6-intel/python2.7-standalone/app/collect
creating /Users/geoffroycouprie/dev/python/tahoe-lafs/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp
creating build/bdist.macosx-10.6-intel/python2.7-standalone/app/lib-dynload
creating build/bdist.macosx-10.6-intel/python2.7-standalone/app/Frameworks
Traceback (most recent call last):
  File "setup.py", line 472, in <module>
    **setup_args
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Library/Python/2.7/site-packages/py2app/build_app.py", line 553, in run
    self._run()
  File "/Library/Python/2.7/site-packages/py2app/build_app.py", line 741, in _run
    self.run_normal()
  File "/Library/Python/2.7/site-packages/py2app/build_app.py", line 812, in run_normal
    mf = self.get_modulefinder()
  File "/Library/Python/2.7/site-packages/py2app/build_app.py", line 691, in get_modulefinder
    debug=debug,
  File "/Library/Python/2.7/site-packages/modulegraph/find_modules.py", line 274, in find_modules
    find_needed_modules(mf, scripts, includes, packages)
  File "/Library/Python/2.7/site-packages/modulegraph/find_modules.py", line 182, in find_needed_modules
    mf.run_script(path)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 673, in run_script
    self.scan_code(co, m)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 1015, in scan_code
    imported_module = self._safe_import_hook(name, m, fromlist, level)[0]
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 917, in _safe_import_hook
    mods = self.import_hook(name, caller, level=level)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 684, in import_hook
    q, tail = self.find_head_package(parent, name, level)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 758, in find_head_package
    q = self.import_module(head, qname, parent)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 840, in import_module
    searchpath, parent)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 1119, in find_module
    fp, buf, stuff = find_module(name, path)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 248, in find_module
    if loader.path.endswith('.py') or loader.path.endswith('.pyw'):
AttributeError: ImpLoader instance has no attribute 'path'

I could try to patch modulegraph to get it working, but I don't want to mess too much my current environment. Can tahoe be built under virtualenv?

[leif]

I've built tahoe using virtualenv 1.9 with pip 1.3, as I describe in #2055.

I don't know if tahoe's setuptools (zetuptools) is invoked by that process.

[zooko]

Here is another reports of a similar-sounding bug:

​https://bitbucket.org/ronaldoussoren/modulegraph/issue/18/attributeerror-imploader-instance-has-no

[geal]

@leif: py2app integrates with setuptools, and it needs modulegraph to recognize the module dependencies of the software.

@zooko: Yes, this is a recent bug. Is there a specific reason why the setuptools version is forced?

[zooko]

Replying to geal:

@zooko: Yes, this is a recent bug. Is there a specific reason why the setuptools version is forced?

What do you mean, "forced"?

[geal]

In setup.py, line 73:

egg = os.path.realpath(glob.glob('setuptools-*.egg')[0])
sys.path.insert(0, egg)
import setuptools; setuptools.bootstrap_install_from = egg

And there is setuptools-0.6c16dev4.egg at the root.

[zooko]

Oh, I see, per comment:14, you think the particular version of setuptools bundled with Tahoe-LAFS is part of the problem. Do you know what versions of pkg_resources have an ImpLoader that is compatible with the latest modulegraph? Per #2044, we are very interested in upgrading our bundled version of setuptools for other reasons (SSL verification of downloads).

[zooko]

Oh, and while you are thinking of that, please notice a related bug that is bothering a user right now: #2079.

[geal]

I use setuptools 1.1.4 with py2app. Maybe that version is a bit too recent, but I'll test it anyway. The documentation for setuptools indicates that upgrade works correctly.

[geal]

Setuptools upgrade

[geal]

Hum sorry, I just thought after the fact that I could send a pull request for the setuptools upgrade. I'll send it right away.

[geal]

So, packaging a .app works, but starting tahoe is "challenging".

It apparently requires a Tahoe.home file (I can directly put it in allmydata-tahoe.app/Contents/), then it needs the tahoe support script, and this one tries to load allmydata-tahoe==1.10.0.post42.dev0.

Could you explain the rationale behind this startup system? It may require some Mac specific tricks to make the packaging work.

[zooko]

Replying to geal:

It apparently requires a Tahoe.home file (I can directly put it in allmydata-tahoe.app/Contents/), then it needs the tahoe support script, and this one tries to load allmydata-tahoe==1.10.0.post42.dev0.

Is it a problem that it is trying to load allmydata-tahoe==1.10.0.post42.dev0 ? That version number is supposed to be automatically generated, so that the version number that the tahoe script is trying to load always exactly matches the version number in the package. Are those numbers different in your setup? If so, we should investigate why.

[geal]

The version number is not a problem. I was just wondering why there was a level of indirection while loading tahoe, instead of just importing the module directly.

Py2app finds the required code by parsing the Python code and finding all the imported modules, so when code is loaded in another way, it must be specified directly in the configuration (it's not always a problem, but some modules can be missed).

If I can get directly to the real entry point (is it src/allmydata/client.py?), I can make the packaging a bit more robust, because py2app will find everything.

[zooko]

Okay, the real entry point is… let's see… in setup.py​:

      entry_points = { 'console_scripts': [ 'tahoe = allmydata.scripts.runner:run' ] },

So the real entry point is allmydata.scripts.runner:run​.

[daira]

geal: there's an alternative entry point in static/tahoe.py​ which is intended for use by static packaging systems like bbfreeze and py2app. It has only been tested recently with bbfreeze.

[daira]

In particular, static/tahoe.py​ explicitly imports all the modules which would otherwise only be imported dynamically by name.

[geal]

I now have a working package :)

I encountered some packaging errors with nevow and formless (they load outside files with pkg_resources), and I removed the check for setuptools to let it launch (I'll see of there's a cleaner way).

Is there a test cluster I can connect to with the client I just launched, to check that everything is ok?

[zooko]

Replying to geal:

I now have a working package :)

Yay!

I encountered some packaging errors with nevow and formless (they load outside files with pkg_resources)

Could you show the packaging errors?

[zooko]

Replying to geal:

I removed the check for setuptools to let it launch (I'll see of there's a cleaner way).

Could you show the problem that prevented it from launching before you removed the check for setuptools?

[zooko]

Replying to geal:

Is there a test cluster I can connect to with the client I just launched, to check that everything is ok?

wiki:TestGrid

[geal]

Replying to zooko:

Could you show the packaging errors?

Here is the stacktrace:

Traceback (most recent call last):
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/__boot__.py", line 43, in <module>
    _run()
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/__boot__.py", line 38, in _run
    exec(compile(source, path, 'exec'), globals(), globals())
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/tahoe.py", line 4, in <module>
    import allmydata
  File "allmydata/__init__.pyc", line 265, in <module>
  File "allmydata/__init__.pyc", line 180, in get_package_versions_and_locations
  File "nevow/__init__.pyc", line 143, in <module>
  File "nevow/__init__.pyc", line 29, in load
  File "nevow/util.pyc", line 183, in _namedAnyWithBuiltinTranslation
  File "twisted/python/reflect.pyc", line 464, in namedAny
  File "twisted/python/reflect.pyc", line 400, in _importAndCheckStack
  File "formless/webform.pyc", line 26, in <module>
  File "build/bdist.macosx-10.6-intel/egg/pkg_resources.py", line 884, in resource_filename
  File "build/bdist.macosx-10.6-intel/egg/pkg_resources.py", line 1516, in get_resource_filename
NotImplementedError: resource_filename() only supported for .egg, not .zip
2013-09-14 15:22:51.592 allmydata-tahoe[11863:707] allmydata-tahoe Error

Py2app stores all the dependencies in site-packages.zip, and pkg_resources has difficulties getting resources from there, but py2app has the 'packages' recipe to move them out of that file, so no problem here.

[geal]

Replying to zooko:

Replying to geal:

I removed the check for setuptools to let it launch (I'll see of there's a cleaner way).

Could you show the problem that prevented it from launching before you removed the check for setuptools?

Here is the stacktrace:

Traceback (most recent call last):
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/__boot__.py", line 43, in <module>
    _run()
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/__boot__.py", line 38, in _run
    exec(compile(source, path, 'exec'), globals(), globals())
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/tahoe.py", line 4, in <module>
    import allmydata
  File "allmydata/__init__.pyc", line 392, in <module>
  File "allmydata/__init__.pyc", line 390, in check_all_requirements
allmydata.PackagingError: 
PackagingError: no version info for setuptools

For debugging purposes, the PYTHONPATH was
  '/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources'
install_requires was
  ['setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface == 3.6.0, == 3.6.1, == 3.6.2, >= 3.6.5', 'Twisted >= 11.0.0', 'foolscap >= 0.6.3', 'pyOpenSSL', 'Nevow >= 0.6.0', 'pycrypto == 2.1.0, == 2.3, >= 2.4.1', 'pyasn1 >= 0.0.8a', 'mock >= 0.8.0', 'pycryptopp >= 0.6.0']
sys.path after importing pkg_resources was
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python27.zip:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/plat-darwin:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/plat-mac:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/plat-mac/lib-scriptpackages:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/lib-tk:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/lib-old:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/lib-dynload:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/site-packages.zip:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/site-packages

And here is the check I removed:

diff --git a/src/allmydata/_auto_deps.py b/src/allmydata/_auto_deps.py
index b7be321..40f4efa 100644
--- a/src/allmydata/_auto_deps.py
+++ b/src/allmydata/_auto_deps.py
@@ -7,7 +7,7 @@
 install_requires = [
     # we require newer versions of setuptools (actually
     # zetuptoolz) to build, but can handle older versions to run
-    "setuptools >= 0.6c6",
+    #"setuptools >= 0.6c6",
 
     "zfec >= 1.1.0",

}}}

[nejucomo]

Note, I just created ticket:2303 which is about acquiring and setting up a dedicated OSX buildslave.

[nejucomo]

We need a new automated packaging test for this work tracked in #2304. When #2303 is complete a buildslave will execute this test.

[daira]

We are unsure whether the .pkg files will be compatible across Mac OS X point releases. For the packaging contract, we intend to just create one .pkg for OS X 10.9 (Mavericks), but also check whether it works on 10.8 in order to document this correctly.

[vu3rdd]

An update on the mac package.

The 182-osx-packaging-5 branch has a make target called 'build-osx-pkg' that builds the pkg and runs a test. I tested the resulting package on OSX 10.9.5 (Mavericks) and in OSX 10.10.x (Yosemite). More testing of both package build and the resultant package with test reports welcome!

[daira]

182-osx-packaging-* has been merged to master. The OS X package still needs wider publicity and testing before we can consider this fixed.

[warner]

FYI, I recently learned that the OS-X "Automator" application can be used to build double-clickable apps which run a shell script. It's a pretty easy process. We could use this to build a "Tahoe" .app file (with icon) which makes sure a client node is running, and then pop open its web UI page. We could probably also build the .app first, then copy a pre-built copy of Tahoe (and dependencies, virtualenv-style) into the .app directory, then bundle the whole thing for distribution.

Some notes: launch Automator.app, choose "Application" from the document-type wizard, drag a Library/Utilities/<Run Shell Script> into the box, set "Pass input:" to as arguments (which lets you drop documents on the new .app and learn their pathnames, might be useful for upload or something), modify the shell script to your heart's content, then Save.

[daira]

We already have a stub double-clickable script in the OS X package, so we don't need another tool to create it. Please open another ticket for the functionality of "making sure that a client node is running and popping open its web UI page". (When writing the specification of this ticket, please don't overspecify implementation!)

[warner]

Incidentally, the OS-X .pkg files for each commit (such as they are), created by make build-osx-pkg, are currently being uploaded to ​https://tahoe-lafs.org/source/tahoe-lafs/tarballs/OS-X-packages/ (by make upload-osx-pkg).