win32 build hell

[zeromus]

I am experiencing hell getting the win32 stuff to build. I should mention that the building works OK by embedding into my python install. But I need a zipfile with an EXE in it if I am going to get any buy-in from my friends.

I am basically following instructions at ​http://tahoe-lafs.org/trac/tahoe-lafs/wiki/AdvancedInstall, but modified in this as a last ditch effort. i assure you the results are the same when i follow those steps as well by running setup.py directly

Here's what I do:

install python-2.5.4.msi to C:\Python25\ install pywin32-214.win32-py2.5.exe extract allmydata-tahoe-1.7.0-SUMO.zip to C:\tahoe-lafs open a command prompt to c:\tahoe-lafs and setup mingw,python25 environment

ran mingw32-make.exe

error: Setup script exited with error: Don't know how to compile cryptopp\x64dll.asm

it seems that distutils msvccompiler.py doesnt know how to compile a .asm file which pycryptopp supplies as a source file to be compiled. it is easy to make msvccompiler.py handle a .asm file. making this change allows me to proceed.. [apparently this has already been covered at ​http://bugs.python.org/issue8597]

build OK. ran mingw32-make.exe windows-exe

build OK. now run windows\dist\tahoe:

Traceback (most recent call last):
  File "tahoe.py", line 1, in <module>
  File "allmydata\__init__.pyc", line 27, in <module>
  File "nevow\__init__.pyc", line 143, in <module>
  File "nevow\__init__.pyc", line 29, in load
  File "nevow\util.pyc", line 183, in _namedAnyWithBuiltinTranslation
  File "twisted\python\reflect.pyc", line 464, in namedAny
  File "twisted\python\reflect.pyc", line 400, in _importAndCheckStack
  File "formless\webform.pyc", line 26, in <module>
  File "pkg_resources.pyc", line 882, in resource_filename
  File "pkg_resources.pyc", line 1344, in get_resource_filename
NotImplementedError: resource_filename() only supported for .egg, not .zip

Around this time, I ought to mention I know diddly squat about python. So I start trying to fix it anyway.

google says (from 5yrs ago) it isnt possible to run resource_filename() from a zip and nevow needs be changed.

newest nevow says setuptools can handle it in a comment on that function

but yet the latest setuptools is what generates the error i pasted. it makes no effort to handle resource_filenames() from a zip.

Now, I can attempt to bully my way through this problem by unzipping library.zip to a directory called library.zip

Then I get this error:

Traceback (most recent call last):
  File "tahoe.py", line 7, in <module>
  File "allmydata\scripts\runner.pyc", line 6, in <module>
  File "C:\tahoe-lafs\windows\dist\library.zip\pkg_resources.pyc", line 666, in
require
  File "C:\tahoe-lafs\windows\dist\library.zip\pkg_resources.pyc", line 565, in
resolve
    equivalent subpath.  For an example namespace handler, see
pkg_resources.DistributionNotFound: twisted

This can be resolved by pasting a bunch of egg directories in the windows\dist\library.zip directory. After I paste twisted, I will get:

pkg_resources.DistributionNotFound?: zope.interface pkg_resources.DistributionNotFound?: setuptools

After those, I will get

pkg_resources.DistributionNotFound?: allmydata-tahoe

This confuses me, because I don't even see my system looking for a file that looks like allmydata-tahoe At this point, all I can do is comment out pkg_resources.require('allmydata-tahoe') from runner.py

Then I will get

C:\tahoe-lafs\windows\dist\library.zip\allmydata\scripts\runner.py:6: UserWarnin
g: Module twisted was already imported from C:\tahoe-lafs\windows\dist\library.z
ip\twisted\__init__.pyc, but c:\tahoe-lafs\windows\dist\library.zip\twisted-10.0
.0-py2.5-win32.egg is being added to sys.path
Traceback (most recent call last):
  File "tahoe.py", line 9, in <module>
TypeError: 'module' object is not callable

Finally, I paste the contents of runner.py into tahoe.py (starting in tahoe.py after _junk = depends)

And then I remove the run function and just replace it with

rc = runner(sys.argv[1:])
sys.exit(rc)

(just making it the main function or whatever you call it in python)

And FINALLY, tahoe.exe works. I can use create-client and configure it to use the public test grid, and then when i use tahoe start...

Can't find twistd (it comes with Twisted). Aborting.

---

flashback hours ago

---

I installed the last public build of Allmydata win32 client. Can't test it now. But look! It comes with tahoe.exe. create-client worked, and then when i use tahoe start...

Can't find twistd (it comes with Twisted).  Aborting.

So I guess I'll try to build my own.

---

Flash forward to present

---

And around that time I gave up.

[zooko]

Wow! Thank you for the detailed ticket. I'll look at it more carefully when I get a chance, but that is not likely to be for at least a week. Hopefully someone else can jump on this and iterate with you before you lose interest and/or forget the details. :-)

I have a couple of very basic question to get started though. I'll post those in a separate comment.

[zooko]

Replying to zeromus:

I am experiencing hell getting the win32 stuff to build. I should mention that the building works OK by embedding into my python install.

What does this mean: "embedding into my python install"? Does that mean you followed ​quickstart.html? Or something else? And did it successfully build so that you had a "tahoe.exe" that you could execute?

[zooko]

Replying to zeromus:

But I need a zipfile with an EXE in it if I am going to get any buy-in from my friends.

See #585 (make it work with bbfreeze) but most of all see #781 (wanted: Windows Packaging Master).

I am basically following instructions at ​http://tahoe-lafs.org/trac/tahoe-lafs/wiki/AdvancedInstall

Wait, there are lots of different instructions in AdvancedInstall: "Installing The Latest Release", "Installing From A Source Tree", "Creating a binary distribution From A Source Tree", "Platform Specific Notes -- Windows", "Platform Specific Notes -- What If That Doesn't Work", and "How Do I Make It Run As A Windows Service". Which one or ones were you following? You described running mingw32-make.exe, but as far as I can tell none of those instructions say to do that.

It is fine not to be following the instructions (I wouldn't be surprised if it turns out that your way is better), but I'm just trying to understand (a) what instructions, if any, you were following, and (b) all of the steps that you took.

i assure you the results are the same when i follow those steps as well by running setup.py directly

What do you mean "running setup.py directly"? Do you mean python setup.py build as instructed on ​quickstart.html? If I understand what happened on your machine correctly, doing python setup.py build would probably incur the same error message:

error: Setup script exited with error: Don't know how to compile cryptopp\x64dll.asm

[zooko]

I opened #1094 (automatically generate binary .egg's of pycryptopp for all of our Supported Platforms) which is about the fact that the build process tried to compile pycryptopp from source instead of acquiring a binary package of pycryptopp.

Also related is ​http://tahoe-lafs.org/trac/pycryptopp/ticket/37 (Win64 and MSVC9 compilation fixes), which is about that fact that when the build process tried to compile pycryptopp from source, it failed to compile.

[zeromus]

Here is a more coherent log of what I did. * following instructions at ​http://tahoe-lafs.org/trac/tahoe-lafs/wiki/AdvancedInstall section "Windows"

1. Download and install Python 2.5.4 from ​http://www.python.org/ftp/python/2.5.4/python-2.5.4.msi.

Installed python-2.5.4.msi to c:\python25

2. Download and install pywin32 from ​http://sourceforge.net/projects/pywin32/files/pywin32/Build%20214/pywin32-214.win32-py2.5.exe/download.

Installed pywin32-214.win32-py2.5.exe to c:\python25

3. Download the latest Tahoe-LAFS release from ​http://tahoe-lafs.org/source/tahoe-lafs/releases/. Unpack it in a convenient place, such as C:\tahoe-lafs.

unpacked ​http://tahoe-lafs.org/source/tahoe-lafs/releases/allmydata-tahoe-1.7.0-SUMO.zip to c:\tahoe-lafs

(In order to build cryptocpp, I need to modify msvccompiler.py by adding .asm to _cpp_extensions)

4. Open a command prompt and cd to the top of the Tahoe-LAFS tree (e.g. cd \tahoe-lafs).

console log: ​http://dl.dropbox.com/u/4260750/bugs/cmd-24-06-2010-214329.txt

That all seems to work. Tahoe runs.

It is interesting that some of these deps compile the second time I try. (Why is it trying to compile x64 asm files anyway on a 32bit platform)

Now, I install py2exe-0.6.9.win32-py2.5.exe and try to run py2exe on the windows stuff to get an exe I can use:

console log: ​http://dl.dropbox.com/u/4260750/bugs/cmd-24-06-2010-215410.txt

* When trying to compile using python 2.6 I get the same errors as 2.5 while trying to compile twisted, but in this case it can't push past it no matter how many times I try. Here is an example

console log: ​http://dl.dropbox.com/u/4260750/temp/cmd-24-06-2010-212013.txt

This is particularly puzzling to me because ticket #1094 says mingw cannot compile extension modules for python 2.6 on windows. But when i begin to build tahoe-lafs using python2.6, it chooses ms visual c++ and then fails to confile conftest.c as part of twisted which requires rpc/rpc.h which is a mingw file.

[zeromus]

It seems that the tahoe.exe included with Allmydata321Setup.exe behaves similarly to the tahoe 1.7.0 py2exe builds I was making. "tahoe start" yields "Can't find twistd (it comes with Twisted). Aborting." while "tahoe run" just exits immediately. It turns out, this is because "tahoe run .\node" doesn't work. There are some docs or bugs somewhere that advise that "tahoe run ." doesn't work in windows--I thought that was strange, but did not expect it to apply to any relative path. Changing the command to "tahoe run c:\path\to\node" causes "tahoe run" to work as expected.

Bug still remains: can't py2exe a tahoe build made with either python25 or python26. Am going to try bbfreeze when i get a chance.

[zooko]

This ticket needs to be split up into a bunch of separate tickets. Some of those separate tickets already exist, notably #1074 (get rid of tahoe.exe launcher). See also the "Windows" tag in the ViewTickets.

[davidsarah]

It is worth testing this again with the 1.8.0 release candidates, which fixed several bugs related to basedir handling.

[aradriel]

I was able to successful build a running version of tahoe-1.8.0 as described on ​http://tahoe-lafs.org/source/tahoe/trunk/docs/quickstart.html and running.html.

Trying building a standalone windows executable via "make windows-exe" as described on ​http://tahoe-lafs.org/trac/tahoe-lafs/wiki/AdvancedInstall under Creating a binary distribution From A Source Tree fails. Errors are (beside version numbers and x64 specific errors) the same as initially portrayed in this ticket by zeromus.

Used tools on my x32 platform:

• Python 2.6.6
  
• py2exe-0.6.9.win32-py2.6
  
• pywin32-214.win32-py2.6
  
• setuptools-0.6c11.win32-py2.6
  
• pyOpenSSL-0.11-py2.6
  
• Win32OpenSSL-1_0_0a
  
• MiniGW Bundle v.20100909
  
• MS Visual Studio 2008 SP1
  

[davidsarah]

To be clear, the problems that might have been fixed are those in comment:6. The workarounds that zeromus needed to get to that point will still be needed.

In any case, current trunk cannot fail with "Can't find twistd (it comes with Twisted). Aborting.", because it calls twisted as a library rather than invoking twistd as a subprocess.

[aradriel]

Thats correct, therefor in 1.8.0-r4803 & current trunk there seems to be a different(?) problem with pkg_resources on calling "make windows-exe":

Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:

    pkg_resources.require("allmydata-tahoe")  # latest installed version
    pkg_resources.require("allmydata-tahoe==1.8.0-r4803")  # this exact version
    pkg_resources.require("allmydata-tahoe>=1.8.0-r4803")  # this version or higher

Note also that the installation directory must be on sys.path at runtime for
this to work.  (e.g. by being the application's script directory, by being on
PYTHONPATH, or by being added to sys.path by your code.)

occurs for every module. heres the full build log:
​http://dl.dropbox.com/u/2929021/build181_w32_exe.txt

So all mentioned modules/packages are not included in Libary.zip anymore.

"python setup.py build" runs smoothly, created a running tahoe script. however, the message from pkg_resources also appears during straight python build.

[davidsarah]

Replying to aradriel:

Thats correct, therefor in 1.8.0-r4803 & current trunk there seems to be a different(?) problem with pkg_resources on calling "make windows-exe":

Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:

    pkg_resources.require("allmydata-tahoe")  # latest installed version
    pkg_resources.require("allmydata-tahoe==1.8.0-r4803")  # this exact version
    pkg_resources.require("allmydata-tahoe>=1.8.0-r4803")  # this version or higher

Note also that the installation directory must be on sys.path at runtime for
this to work.  (e.g. by being the application's script directory, by being on
PYTHONPATH, or by being added to sys.path by your code.)

occurs for every module.

Those are warnings, not errors. (I know that's not clear from the message.) They are caused by the change in #530, which I think should probably be reverted.

heres the full build log:
​http://dl.dropbox.com/u/2929021/build181_w32_exe.txt

So all mentioned modules/packages are not included in Libary.zip anymore.

If they aren't included, I think it's probably for some other reason than the use of --multi-version.

[zooko]

pywin32 is no longer used by Tahoe-LAFS, so that should simplify this a little. What else is left to do for this ticket?

[zooko]

duplicate of #195