# win32 build hell

Reported by: Owned by: zeromus somebody major soon packaging 1.8.0 windows setuptools pycryptopp nevow twisted freestorm77@…, aradriel@…

I am experiencing hell getting the win32 stuff to build. I should mention that the building works OK by embedding into my python install. But I need a zipfile with an EXE in it if I am going to get any buy-in from my friends.

I am basically following instructions at http://tahoe-lafs.org/trac/tahoe-lafs/wiki/AdvancedInstall, but modified in this as a last ditch effort. i assure you the results are the same when i follow those steps as well by running setup.py directly

Here's what I do:

install python-2.5.4.msi to C:\Python25\ install pywin32-214.win32-py2.5.exe extract allmydata-tahoe-1.7.0-SUMO.zip to C:\tahoe-lafs open a command prompt to c:\tahoe-lafs and setup mingw,python25 environment

ran mingw32-make.exe

error: Setup script exited with error: Don't know how to compile cryptopp\x64dll.asm


it seems that distutils msvccompiler.py doesnt know how to compile a .asm file which pycryptopp supplies as a source file to be compiled. it is easy to make msvccompiler.py handle a .asm file. making this change allows me to proceed.. [apparently this has already been covered at http://bugs.python.org/issue8597]

build OK. ran mingw32-make.exe windows-exe

build OK. now run windows\dist\tahoe:

Traceback (most recent call last):
File "tahoe.py", line 1, in <module>
File "allmydata\__init__.pyc", line 27, in <module>
File "nevow\__init__.pyc", line 143, in <module>
File "nevow\__init__.pyc", line 29, in load
File "nevow\util.pyc", line 183, in _namedAnyWithBuiltinTranslation
File "twisted\python\reflect.pyc", line 464, in namedAny
File "twisted\python\reflect.pyc", line 400, in _importAndCheckStack
File "formless\webform.pyc", line 26, in <module>
File "pkg_resources.pyc", line 882, in resource_filename
File "pkg_resources.pyc", line 1344, in get_resource_filename
NotImplementedError: resource_filename() only supported for .egg, not .zip


Around this time, I ought to mention I know diddly squat about python. So I start trying to fix it anyway.

google says (from 5yrs ago) it isnt possible to run resource_filename() from a zip and nevow needs be changed.

newest nevow says setuptools can handle it in a comment on that function

but yet the latest setuptools is what generates the error i pasted. it makes no effort to handle resource_filenames() from a zip.

Now, I can attempt to bully my way through this problem by unzipping library.zip to a directory called library.zip

Then I get this error:

Traceback (most recent call last):
File "tahoe.py", line 7, in <module>
File "allmydata\scripts\runner.pyc", line 6, in <module>
File "C:\tahoe-lafs\windows\dist\library.zip\pkg_resources.pyc", line 666, in
require
File "C:\tahoe-lafs\windows\dist\library.zip\pkg_resources.pyc", line 565, in
resolve
equivalent subpath.  For an example namespace handler, see
pkg_resources.DistributionNotFound: twisted


This can be resolved by pasting a bunch of egg directories in the windows\dist\library.zip directory. After I paste twisted, I will get:

pkg_resources.DistributionNotFound?: zope.interface pkg_resources.DistributionNotFound?: setuptools

After those, I will get

pkg_resources.DistributionNotFound?: allmydata-tahoe

This confuses me, because I don't even see my system looking for a file that looks like allmydata-tahoe At this point, all I can do is comment out pkg_resources.require('allmydata-tahoe') from runner.py

Then I will get

C:\tahoe-lafs\windows\dist\library.zip\allmydata\scripts\runner.py:6: UserWarnin
g: Module twisted was already imported from C:\tahoe-lafs\windows\dist\library.z
ip\twisted\__init__.pyc, but c:\tahoe-lafs\windows\dist\library.zip\twisted-10.0
.0-py2.5-win32.egg is being added to sys.path
Traceback (most recent call last):
File "tahoe.py", line 9, in <module>
TypeError: 'module' object is not callable


Finally, I paste the contents of runner.py into tahoe.py (starting in tahoe.py after _junk = depends)

And then I remove the run function and just replace it with

rc = runner(sys.argv[1:])
sys.exit(rc)


(just making it the main function or whatever you call it in python)

And FINALLY, tahoe.exe works. I can use create-client and configure it to use the public test grid, and then when i use tahoe start...

Can't find twistd (it comes with Twisted). Aborting.

flashback hours ago

I installed the last public build of Allmydata win32 client. Can't test it now. But look! It comes with tahoe.exe. create-client worked, and then when i use tahoe start...

Can't find twistd (it comes with Twisted).  Aborting.


So I guess I'll try to build my own.

Flash forward to present

And around that time I gave up.

### comment:1 Changed at 2010-06-21T20:52:34Z by zooko

• Component changed from unknown to packaging
• Keywords windows setuptools pycryptopp nevow twisted added
• Milestone changed from undecided to 1.7.1
• Owner changed from nobody to somebody

Wow! Thank you for the detailed ticket. I'll look at it more carefully when I get a chance, but that is not likely to be for at least a week. Hopefully someone else can jump on this and iterate with you before you lose interest and/or forget the details. :-)

I have a couple of very basic question to get started though. I'll post those in a separate comment.

### comment:2 in reply to: ↑ description Changed at 2010-06-21T20:54:40Z by zooko

I am experiencing hell getting the win32 stuff to build. I should mention that the building works OK by embedding into my python install.

What does this mean: "embedding into my python install"? Does that mean you followed quickstart.html? Or something else? And did it successfully build so that you had a "tahoe.exe" that you could execute?

### comment:3 in reply to: ↑ description Changed at 2010-06-21T21:18:14Z by zooko

But I need a zipfile with an EXE in it if I am going to get any buy-in from my friends.

See #585 (make it work with bbfreeze) but most of all see #781 (wanted: Windows Packaging Master).

I am basically following instructions at http://tahoe-lafs.org/trac/tahoe-lafs/wiki/AdvancedInstall

Wait, there are lots of different instructions in AdvancedInstall: "Installing The Latest Release", "Installing From A Source Tree", "Creating a binary distribution From A Source Tree", "Platform Specific Notes -- Windows", "Platform Specific Notes -- What If That Doesn't Work", and "How Do I Make It Run As A Windows Service". Which one or ones were you following? You described running mingw32-make.exe, but as far as I can tell none of those instructions say to do that.

It is fine not to be following the instructions (I wouldn't be surprised if it turns out that your way is better), but I'm just trying to understand (a) what instructions, if any, you were following, and (b) all of the steps that you took.

i assure you the results are the same when i follow those steps as well by running setup.py directly

What do you mean "running setup.py directly"? Do you mean python setup.py build as instructed on quickstart.html? If I understand what happened on your machine correctly, doing python setup.py build would probably incur the same error message:

error: Setup script exited with error: Don't know how to compile cryptopp\x64dll.asm


### comment:4 Changed at 2010-06-21T22:17:16Z by zooko

I opened #1094 (automatically generate binary .egg's of pycryptopp for all of our Supported Platforms) which is about the fact that the build process tried to compile pycryptopp from source instead of acquiring a binary package of pycryptopp.

Also related is http://tahoe-lafs.org/trac/pycryptopp/ticket/37 (Win64 and MSVC9 compilation fixes), which is about that fact that when the build process tried to compile pycryptopp from source, it failed to compile.

### comment:5 Changed at 2010-06-25T03:21:01Z by zeromus

Here is a more coherent log of what I did. * following instructions at http://tahoe-lafs.org/trac/tahoe-lafs/wiki/AdvancedInstall section "Windows"

Installed python-2.5.4.msi to c:\python25

Installed pywin32-214.win32-py2.5.exe to c:\python25

1. Download the latest Tahoe-LAFS release from http://tahoe-lafs.org/source/tahoe-lafs/releases/. Unpack it in a convenient place, such as C:\tahoe-lafs.

unpacked http://tahoe-lafs.org/source/tahoe-lafs/releases/allmydata-tahoe-1.7.0-SUMO.zip to c:\tahoe-lafs

(In order to build cryptocpp, I need to modify msvccompiler.py by adding .asm to _cpp_extensions)

1. Open a command prompt and cd to the top of the Tahoe-LAFS tree (e.g. cd \tahoe-lafs).

That all seems to work. Tahoe runs.

It is interesting that some of these deps compile the second time I try. (Why is it trying to compile x64 asm files anyway on a 32bit platform)

Now, I install py2exe-0.6.9.win32-py2.5.exe and try to run py2exe on the windows stuff to get an exe I can use:

* When trying to compile using python 2.6 I get the same errors as 2.5 while trying to compile twisted, but in this case it can't push past it no matter how many times I try. Here is an example

This is particularly puzzling to me because ticket #1094 says mingw cannot compile extension modules for python 2.6 on windows. But when i begin to build tahoe-lafs using python2.6, it chooses ms visual c++ and then fails to confile conftest.c as part of twisted which requires rpc/rpc.h which is a mingw file.

### comment:6 Changed at 2010-06-27T23:30:19Z by zeromus

It seems that the tahoe.exe included with Allmydata321Setup.exe behaves similarly to the tahoe 1.7.0 py2exe builds I was making. "tahoe start" yields "Can't find twistd (it comes with Twisted). Aborting." while "tahoe run" just exits immediately. It turns out, this is because "tahoe run .\node" doesn't work. There are some docs or bugs somewhere that advise that "tahoe run ." doesn't work in windows--I thought that was strange, but did not expect it to apply to any relative path. Changing the command to "tahoe run -C c:\path\to\node" causes "tahoe run" to work as expected.

Bug still remains: can't py2exe a tahoe build made with either python25 or python26. Am going to try bbfreeze when i get a chance.

Last edited at 2010-06-27T23:46:02Z by zeromus (previous) (diff)

### comment:8 Changed at 2010-07-17T07:36:52Z by zooko

This ticket needs to be split up into a bunch of separate tickets. Some of those separate tickets already exist, notably #1074 (get rid of tahoe.exe launcher). See also the "Windows" tag in the ViewTickets.

### comment:9 Changed at 2010-07-18T02:35:40Z by davidsarah

• Milestone changed from 1.7.1 to soon

### comment:10 follow-up: ↓ 12 Changed at 2010-08-27T20:04:16Z by davidsarah

It is worth testing this again with the 1.8.0 release candidates, which fixed several bugs related to basedir handling.

### comment:11 Changed at 2010-11-13T23:11:21Z by aradriel

• Version changed from 1.7.0 to 1.8.0

I was able to successful build a running version of tahoe-1.8.0 as described on http://tahoe-lafs.org/source/tahoe/trunk/docs/quickstart.html and running.html.

Trying building a standalone windows executable via "make windows-exe" as described on http://tahoe-lafs.org/trac/tahoe-lafs/wiki/AdvancedInstall under Creating a binary distribution From A Source Tree fails. Errors are (beside version numbers and x64 specific errors) the same as initially portrayed in this ticket by zeromus.

Used tools on my x32 platform:

• Python 2.6.6
• py2exe-0.6.9.win32-py2.6
• pywin32-214.win32-py2.6
• setuptools-0.6c11.win32-py2.6
• pyOpenSSL-0.11-py2.6
• Win32OpenSSL-1_0_0a
• MiniGW Bundle v.20100909
• MS Visual Studio 2008 SP1

### comment:12 in reply to: ↑ 10 ; follow-up: ↓ 13 Changed at 2010-11-14T00:52:06Z by davidsarah

To be clear, the problems that might have been fixed are those in comment:6. The workarounds that zeromus needed to get to that point will still be needed.

In any case, current trunk cannot fail with "Can't find twistd (it comes with Twisted). Aborting.", because it calls twisted as a library rather than invoking twistd as a subprocess.

### comment:13 in reply to: ↑ 12 ; follow-up: ↓ 14 Changed at 2010-11-14T15:56:49Z by aradriel

Thats correct, therefor in 1.8.0-r4803 & current trunk there seems to be a different(?) problem with pkg_resources on calling "make windows-exe":

Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:

pkg_resources.require("allmydata-tahoe")  # latest installed version
pkg_resources.require("allmydata-tahoe==1.8.0-r4803")  # this exact version
pkg_resources.require("allmydata-tahoe>=1.8.0-r4803")  # this version or higher

Note also that the installation directory must be on sys.path at runtime for
this to work.  (e.g. by being the application's script directory, by being on


occurs for every module. heres the full build log:
http://dl.dropbox.com/u/2929021/build181_w32_exe.txt

So all mentioned modules/packages are not included in Libary.zip anymore.

"python setup.py build" runs smoothly, created a running tahoe script. however, the message from pkg_resources also appears during straight python build.

### comment:14 in reply to: ↑ 13 Changed at 2010-11-15T00:17:55Z by davidsarah

Thats correct, therefor in 1.8.0-r4803 & current trunk there seems to be a different(?) problem with pkg_resources on calling "make windows-exe":

Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:

pkg_resources.require("allmydata-tahoe")  # latest installed version
pkg_resources.require("allmydata-tahoe==1.8.0-r4803")  # this exact version
pkg_resources.require("allmydata-tahoe>=1.8.0-r4803")  # this version or higher

Note also that the installation directory must be on sys.path at runtime for
this to work.  (e.g. by being the application's script directory, by being on


occurs for every module.

Those are warnings, not errors. (I know that's not clear from the message.) They are caused by the change in #530, which I think should probably be reverted.

heres the full build log:
http://dl.dropbox.com/u/2929021/build181_w32_exe.txt

So all mentioned modules/packages are not included in Libary.zip anymore.

If they aren't included, I think it's probably for some other reason than the use of --multi-version.

### comment:15 Changed at 2013-05-03T22:01:48Z by zooko

pywin32 is no longer used by Tahoe-LAFS, so that should simplify this a little. What else is left to do for this ticket?

### comment:16 Changed at 2014-09-25T16:02:27Z by zooko

• Description modified (diff)
• Resolution set to duplicate
• Status changed from new to closed

duplicate of #195

Note: See TracTickets for help on using tickets.