#182 new enhancement

build a .pkg installer for Mac OS X 10.9 Mavericks (intel-x86-64)

Reported by: zooko Owned by: zooko
Priority: major Milestone: soon
Component: packaging Version: 0.6.1
Keywords: mac usability install openitp-packaging Cc: arch_o_median, drewp@…
Launchpad Bug:

Description (last modified by zooko)

My Mom wants to see some home movies of my children playing in the back yard. Even though our docs/quickstart.rst is very well written, it is still far beyond her ken to build Tahoe from source. How could she get Tahoe installed on her home computer, which is a PowerPC G5 running Mac OS 10.3.9?

One way is that I could upload a binary .egg and instruct her in how to install Python 2.5.1 and easy_install and then how to run "easy_install allmydata-tahoe". However, the dependent libraries like Nevow and Twisted and so forth wouldn't get automatically installed by easy_install so that wouldn't quite work.

Another way is that I could build binary .egg packages of all of our dependent libraries and host them somewhere or bundle them into the tahoe binary .egg.

Another way is that we could use the "py2app" feature to build an installer than works on Mac OS 10.

Attachments (1)

0001-use-a-recent-version-of-setuptools.patch (419.9 KB) - added by geal at 2013-09-12T12:32:45Z.
Setuptools upgrade

Download all attachments as: .zip

Change History (49)

comment:1 Changed at 2007-10-16T02:35:07Z by zooko

The py2app approach still requires that we bundle binaries of all of our dependent libraries.

comment:2 Changed at 2007-10-19T23:13:29Z by zooko

So perhaps a next-step on this is to experiment with making a complete bundle of Tahoe and all dependencies. :-) I did that for Zfone and then used py2app to build a Mac .app of the result.

comment:3 Changed at 2007-10-22T22:20:01Z by warner

The py2app approach would require those dependent libraries to be present on the package-builder's machine, but the resulting package would be completely standalone, right?

I've started playing with py2app and have run into some problems (it seems to refuse to include foolscap in the bundle, despite being explicitly imported by the main entry-point script, even though it includes zfec and twisted just fine). But overall I think it will work.

So my main question is: what exactly should this packaged application look like?

One possibility:

  • the tahoe node is created by the install process (using one of those fancy Assistants)
  • the node is *not* started at boot time
  • when the user double-clicks on the tahoe icon, their node is started, and a small window is created
    • this window has a button which, when pressed, sends their web browser to the local start page
    • dropping a URI in the window sends the web browser to the corresponding /URI/NNN page
    • dropping a file in the window causes the file to be uploaded, and the URI is displayed in a way that allows it to be cut-and-pasted into another application

thoughts?

comment:4 Changed at 2007-10-22T22:57:31Z by zooko

The py2app approach would require those dependent libraries to be present on the package-builder's machine, but the resulting package would be completely standalone, right?

Right.

the tahoe node is created by the install process (using one of those fancy Assistants)

A Mac OS X idiom which I love is that installing is merely dragging and dropping (i.e. copying or moving) the Thingie.app into your filesystem. This pattern abjures running an "installer" (I'm not sure if that's related to an "Assistant") at install-time. I hope we use this pattern!

That would imply that node-creation happens at runtime instead, for example "if there is not already a node in ~/.tahoe".

comment:5 Changed at 2007-12-21T21:35:24Z by zooko

  • Cc arch_o_median added
  • Milestone changed from undecided to 0.7.2

comment:6 Changed at 2008-01-05T03:50:45Z by warner

  • Milestone changed from 0.7.2 to 0.9.0

comment:7 Changed at 2008-01-30T16:59:33Z by terrell

additionally, there should be a prominent place for picking which network to be a part of (friendnet, testgrid, allmydata public grid, etc). Is that a dropdown with the easy choices? A means to discover their friend's introducer.furl?

That would imply that node-creation happens at runtime instead, for example "if there is not already a node in ~/.tahoe".

yes, totally.

later, create some kind of manager/preferences for multiple nodes, but initially, just create the one if it doesn't exist. if it does already exist, have a suppressible popup that would allow an overwrite/cleanstart.

comment:8 Changed at 2008-03-08T04:14:54Z by zooko

  • Milestone changed from 0.9.0 (Allmydata 3.0 final) to 0.10.0

comment:9 Changed at 2008-05-29T22:26:06Z by warner

  • Milestone changed from 1.1.0 to 1.2.0

comment:10 Changed at 2009-01-25T01:30:53Z by drewp

  • Cc drewp@… added

comment:11 Changed at 2009-06-30T12:39:17Z by zooko

  • Milestone changed from 1.5.0 to eventually

comment:12 Changed at 2009-10-28T07:12:55Z by davidsarah

  • Keywords usability added

comment:13 Changed at 2009-12-12T02:37:32Z by davidsarah

  • Keywords install added

comment:14 Changed at 2013-09-11T21:30:14Z by geal

  • Description modified (diff)

Hi,

I can help a bit on this bug. I have already messed with py2app to generate .app packages and I have some code to automatically generate DMG images.

I first tried to add some basic options to py2app and ran it, to see if everything is ok. Unfortunately, it seems the latest version of modulegraph (necessary for py2app) is in conflict with the setuptools version that is used in the project (ImpLoader? is part of pkg_resources).

running py2app
running build_py
copying src/allmydata/_version.py -> build/bdist.macosx-10.6-intel/lib/allmydata
creating /Users/geoffroycouprie/dev/python/tahoe-lafs/build/bdist.macosx-10.6-intel/python2.7-standalone/app
creating /Users/geoffroycouprie/dev/python/tahoe-lafs/build/bdist.macosx-10.6-intel/python2.7-standalone/app/collect
creating /Users/geoffroycouprie/dev/python/tahoe-lafs/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp
creating build/bdist.macosx-10.6-intel/python2.7-standalone/app/lib-dynload
creating build/bdist.macosx-10.6-intel/python2.7-standalone/app/Frameworks
Traceback (most recent call last):
  File "setup.py", line 472, in <module>
    **setup_args
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Library/Python/2.7/site-packages/py2app/build_app.py", line 553, in run
    self._run()
  File "/Library/Python/2.7/site-packages/py2app/build_app.py", line 741, in _run
    self.run_normal()
  File "/Library/Python/2.7/site-packages/py2app/build_app.py", line 812, in run_normal
    mf = self.get_modulefinder()
  File "/Library/Python/2.7/site-packages/py2app/build_app.py", line 691, in get_modulefinder
    debug=debug,
  File "/Library/Python/2.7/site-packages/modulegraph/find_modules.py", line 274, in find_modules
    find_needed_modules(mf, scripts, includes, packages)
  File "/Library/Python/2.7/site-packages/modulegraph/find_modules.py", line 182, in find_needed_modules
    mf.run_script(path)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 673, in run_script
    self.scan_code(co, m)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 1015, in scan_code
    imported_module = self._safe_import_hook(name, m, fromlist, level)[0]
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 917, in _safe_import_hook
    mods = self.import_hook(name, caller, level=level)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 684, in import_hook
    q, tail = self.find_head_package(parent, name, level)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 758, in find_head_package
    q = self.import_module(head, qname, parent)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 840, in import_module
    searchpath, parent)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 1119, in find_module
    fp, buf, stuff = find_module(name, path)
  File "/Library/Python/2.7/site-packages/modulegraph/modulegraph.py", line 248, in find_module
    if loader.path.endswith('.py') or loader.path.endswith('.pyw'):
AttributeError: ImpLoader instance has no attribute 'path'

I could try to patch modulegraph to get it working, but I don't want to mess too much my current environment. Can tahoe be built under virtualenv?

Version 0, edited at 2013-09-11T21:30:14Z by geal (next)

comment:15 Changed at 2013-09-11T21:48:30Z by leif

I've built tahoe using virtualenv 1.9 with pip 1.3, as I describe in #2055.

I don't know if tahoe's setuptools (zetuptools) is invoked by that process.

comment:17 follow-up: Changed at 2013-09-12T09:45:29Z by geal

@leif: py2app integrates with setuptools, and it needs modulegraph to recognize the module dependencies of the software.

@zooko: Yes, this is a recent bug. Is there a specific reason why the setuptools version is forced?

comment:18 in reply to: ↑ 17 Changed at 2013-09-12T11:14:32Z by zooko

  • Description modified (diff)

Replying to geal:

@zooko: Yes, this is a recent bug. Is there a specific reason why the setuptools version is forced?

What do you mean, "forced"?

comment:19 follow-up: Changed at 2013-09-12T11:27:09Z by geal

In setup.py, line 73:

egg = os.path.realpath(glob.glob('setuptools-*.egg')[0])
sys.path.insert(0, egg)
import setuptools; setuptools.bootstrap_install_from = egg

And there is setuptools-0.6c16dev4.egg at the root.

comment:20 in reply to: ↑ 19 Changed at 2013-09-12T11:29:02Z by zooko

Oh, I see, per comment:14, you think the particular version of setuptools bundled with Tahoe-LAFS is part of the problem. Do you know what versions of pkg_resources have an ImpLoader that is compatible with the latest modulegraph? Per #2044, we are very interested in upgrading our bundled version of setuptools for other reasons (SSL verification of downloads).

comment:21 Changed at 2013-09-12T11:30:01Z by zooko

Oh, and while you are thinking of that, please notice a related bug that is bothering a user right now: #2079.

comment:22 Changed at 2013-09-12T11:54:22Z by geal

I use setuptools 1.1.4 with py2app. Maybe that version is a bit too recent, but I'll test it anyway. The documentation for setuptools indicates that upgrade works correctly.

Changed at 2013-09-12T12:32:45Z by geal

Setuptools upgrade

comment:23 Changed at 2013-09-12T12:34:25Z by geal

Hum sorry, I just thought after the fact that I could send a pull request for the setuptools upgrade. I'll send it right away.

comment:24 follow-up: Changed at 2013-09-12T14:01:32Z by geal

So, packaging a .app works, but starting tahoe is "challenging".

It apparently requires a Tahoe.home file (I can directly put it in allmydata-tahoe.app/Contents/), then it needs the tahoe support script, and this one tries to load allmydata-tahoe==1.10.0.post42.dev0.

Could you explain the rationale behind this startup system? It may require some Mac specific tricks to make the packaging work.

comment:25 in reply to: ↑ 24 Changed at 2013-09-12T14:14:31Z by zooko

Replying to geal:

It apparently requires a Tahoe.home file (I can directly put it in allmydata-tahoe.app/Contents/), then it needs the tahoe support script, and this one tries to load allmydata-tahoe==1.10.0.post42.dev0.

Is it a problem that it is trying to load allmydata-tahoe==1.10.0.post42.dev0 ? That version number is supposed to be automatically generated, so that the version number that the tahoe script is trying to load always exactly matches the version number in the package. Are those numbers different in your setup? If so, we should investigate why.

comment:26 follow-up: Changed at 2013-09-12T14:28:51Z by geal

The version number is not a problem. I was just wondering why there was a level of indirection while loading tahoe, instead of just importing the module directly.

Py2app finds the required code by parsing the Python code and finding all the imported modules, so when code is loaded in another way, it must be specified directly in the configuration (it's not always a problem, but some modules can be missed).

If I can get directly to the real entry point (is it src/allmydata/client.py?), I can make the packaging a bit more robust, because py2app will find everything.

comment:27 in reply to: ↑ 26 Changed at 2013-09-12T14:50:55Z by zooko

Okay, the real entry point is… let's see… in setup.py:

      entry_points = { 'console_scripts': [ 'tahoe = allmydata.scripts.runner:run' ] },

So the real entry point is allmydata.scripts.runner:run.

comment:28 Changed at 2013-09-12T15:07:48Z by daira

geal: there's an alternative entry point in static/tahoe.py which is intended for use by static packaging systems like bbfreeze and py2app. It has only been tested recently with bbfreeze.

comment:29 Changed at 2013-09-12T15:10:10Z by daira

In particular, static/tahoe.py explicitly imports all the modules which would otherwise only be imported dynamically by name.

comment:30 follow-ups: Changed at 2013-09-12T16:43:38Z by geal

I now have a working package :)

I encountered some packaging errors with nevow and formless (they load outside files with pkg_resources), and I removed the check for setuptools to let it launch (I'll see of there's a cleaner way).

Is there a test cluster I can connect to with the client I just launched, to check that everything is ok?

Last edited at 2013-09-12T16:43:51Z by geal (previous) (diff)

comment:31 in reply to: ↑ 30 ; follow-up: Changed at 2013-09-12T17:34:04Z by zooko

Replying to geal:

I now have a working package :)

Yay!

I encountered some packaging errors with nevow and formless (they load outside files with pkg_resources)

Could you show the packaging errors?

comment:32 in reply to: ↑ 30 ; follow-up: Changed at 2013-09-12T17:34:48Z by zooko

Replying to geal:

I removed the check for setuptools to let it launch (I'll see of there's a cleaner way).

Could you show the problem that prevented it from launching before you removed the check for setuptools?

comment:33 in reply to: ↑ 30 Changed at 2013-09-12T17:35:36Z by zooko

Replying to geal:

Is there a test cluster I can connect to with the client I just launched, to check that everything is ok?

wiki:TestGrid

comment:34 in reply to: ↑ 31 Changed at 2013-09-14T13:25:47Z by geal

Replying to zooko:

Could you show the packaging errors?

Here is the stacktrace:

Traceback (most recent call last):
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/__boot__.py", line 43, in <module>
    _run()
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/__boot__.py", line 38, in _run
    exec(compile(source, path, 'exec'), globals(), globals())
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/tahoe.py", line 4, in <module>
    import allmydata
  File "allmydata/__init__.pyc", line 265, in <module>
  File "allmydata/__init__.pyc", line 180, in get_package_versions_and_locations
  File "nevow/__init__.pyc", line 143, in <module>
  File "nevow/__init__.pyc", line 29, in load
  File "nevow/util.pyc", line 183, in _namedAnyWithBuiltinTranslation
  File "twisted/python/reflect.pyc", line 464, in namedAny
  File "twisted/python/reflect.pyc", line 400, in _importAndCheckStack
  File "formless/webform.pyc", line 26, in <module>
  File "build/bdist.macosx-10.6-intel/egg/pkg_resources.py", line 884, in resource_filename
  File "build/bdist.macosx-10.6-intel/egg/pkg_resources.py", line 1516, in get_resource_filename
NotImplementedError: resource_filename() only supported for .egg, not .zip
2013-09-14 15:22:51.592 allmydata-tahoe[11863:707] allmydata-tahoe Error

Py2app stores all the dependencies in site-packages.zip, and pkg_resources has difficulties getting resources from there, but py2app has the 'packages' recipe to move them out of that file, so no problem here.

comment:35 in reply to: ↑ 32 Changed at 2013-09-14T13:28:49Z by geal

Replying to zooko:

Replying to geal:

I removed the check for setuptools to let it launch (I'll see of there's a cleaner way).

Could you show the problem that prevented it from launching before you removed the check for setuptools?

Here is the stacktrace:

Traceback (most recent call last):
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/__boot__.py", line 43, in <module>
    _run()
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/__boot__.py", line 38, in _run
    exec(compile(source, path, 'exec'), globals(), globals())
  File "/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/tahoe.py", line 4, in <module>
    import allmydata
  File "allmydata/__init__.pyc", line 392, in <module>
  File "allmydata/__init__.pyc", line 390, in check_all_requirements
allmydata.PackagingError: 
PackagingError: no version info for setuptools

For debugging purposes, the PYTHONPATH was
  '/Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources'
install_requires was
  ['setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface == 3.6.0, == 3.6.1, == 3.6.2, >= 3.6.5', 'Twisted >= 11.0.0', 'foolscap >= 0.6.3', 'pyOpenSSL', 'Nevow >= 0.6.0', 'pycrypto == 2.1.0, == 2.3, >= 2.4.1', 'pyasn1 >= 0.0.8a', 'mock >= 0.8.0', 'pycryptopp >= 0.6.0']
sys.path after importing pkg_resources was
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python27.zip:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/plat-darwin:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/plat-mac:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/plat-mac/lib-scriptpackages:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/lib-tk:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/lib-old:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/lib-dynload:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/site-packages.zip:
  /Users/geoffroycouprie/dev/python/tahoe-lafs/dist/allmydata-tahoe.app/Contents/Resources/lib/python2.7/site-packages

And here is the check I removed:

diff --git a/src/allmydata/_auto_deps.py b/src/allmydata/_auto_deps.py
index b7be321..40f4efa 100644
--- a/src/allmydata/_auto_deps.py
+++ b/src/allmydata/_auto_deps.py
@@ -7,7 +7,7 @@
 install_requires = [
     # we require newer versions of setuptools (actually
     # zetuptoolz) to build, but can handle older versions to run
-    "setuptools >= 0.6c6",
+    #"setuptools >= 0.6c6",
 
     "zfec >= 1.1.0",

}}}

comment:36 Changed at 2014-09-08T16:29:35Z by daira

  • Keywords openitp-packaging added
  • Owner changed from somebody to vu3rdd

comment:37 Changed at 2014-09-25T15:56:16Z by daira

  • Milestone changed from eventually to soon

comment:38 Changed at 2014-09-25T16:23:16Z by daira

  • Summary changed from user-friendly installer for Mac -- for my Mom! to build a .pkg installer for Mac OS X 10.9

comment:39 Changed at 2014-09-25T16:38:14Z by daira

  • Summary changed from build a .pkg installer for Mac OS X 10.9 to build a .pkg installer for Mac OS X 10.9 Mavericks (intel-x86-64)

comment:40 Changed at 2014-09-29T16:48:06Z by nejucomo

Note, I just created #2303 which is about acquiring and setting up a dedicated OSX buildslave.

Last edited at 2014-09-29T17:09:05Z by nejucomo (previous) (diff)

comment:41 Changed at 2014-09-29T17:08:33Z by nejucomo

We need a new automated packaging test for this work tracked in #2304. When #2303 is complete a buildslave will execute this test.

comment:42 Changed at 2014-09-29T17:11:07Z by daira

We are unsure whether the .pkg files will be compatible across Mac OS X point releases. For the packaging contract, we intend to just create one .pkg for OS X 10.9 (Mavericks), but also check whether it works on 10.8 in order to document this correctly.

comment:43 Changed at 2014-10-20T16:11:09Z by vu3rdd

An update on the mac package.

The 182-osx-packaging-5 branch has a make target called 'build-osx-pkg' that builds the pkg and runs a test. I tested the resulting package on OSX 10.9.5 (Mavericks) and in OSX 10.10.x (Yosemite). More testing of both package build and the resultant package with test reports welcome!

comment:44 Changed at 2014-10-28T19:08:21Z by daira

182-osx-packaging-* has been merged to master. The OS X package still needs wider publicity and testing before we can consider this fixed.

Last edited at 2014-10-28T19:08:39Z by daira (previous) (diff)

comment:45 Changed at 2014-10-28T19:08:56Z by daira

  • Owner changed from vu3rdd to zooko

comment:46 Changed at 2015-05-08T18:50:07Z by warner

FYI, I recently learned that the OS-X "Automator" application can be used to build double-clickable apps which run a shell script. It's a pretty easy process. We could use this to build a "Tahoe" .app file (with icon) which makes sure a client node is running, and then pop open its web UI page. We could probably also build the .app first, then copy a pre-built copy of Tahoe (and dependencies, virtualenv-style) into the .app directory, then bundle the whole thing for distribution.

Some notes: launch Automator.app, choose "Application" from the document-type wizard, drag a Library/Utilities/<Run Shell Script> into the box, set "Pass input:" to as arguments (which lets you drop documents on the new .app and learn their pathnames, might be useful for upload or something), modify the shell script to your heart's content, then Save.

comment:47 Changed at 2015-05-09T23:59:40Z by daira

We already have a stub double-clickable script in the OS X package, so we don't need another tool to create it. Please open another ticket for the functionality of "making sure that a client node is running and popping open its web UI page". (When writing the specification of this ticket, please don't overspecify implementation!)

comment:48 Changed at 2015-07-16T18:21:34Z by warner

Incidentally, the OS-X .pkg files for each commit (such as they are), created by make build-osx-pkg, are currently being uploaded to https://tahoe-lafs.org/source/tahoe-lafs/tarballs/OS-X-packages/ (by make upload-osx-pkg).

Note: See TracTickets for help on using tickets.