organize the documentation for installation on Windows

[zooko]

Before the official Tahoe-LAFS v1.5.0 release, I would like for someone to follow the official install doc (​http://allmydata.org/source/tahoe/trunk/docs/install.html ) on a Windows system. If you've never installed Tahoe-LAFS on your Windows system before, all the better! We have automated tests of build and certain install steps on Windows (​http://allmydata.org/buildbot/builders/windows , ​http://allmydata.org/buildbot/builders/BlackDew%20windows-mingw32-py26 ) and cygwin (​http://allmydata.org/buildbot/builders/cygwin ), but I would feel better if an actual human tried it.

Hm, say Black Dew, why is your Windows buildslave off-line?

[bdew]

I was away for few days and it looks like it's (crappy) internet connection is half-dead, i'll try to fix it later today.

[zooko]

Nobody has yet succeeded at installing the current snapshot (release candidate for v1.5.0) Tahoe-LAFS from source on Windows. But we're getting closer!

​http://allmydata.org/pipermail/tahoe-dev/2009-July/002426.html

[swillden]

I have succeeded, but I'm not sure exactly which steps were required. I'm repeating the process in a clean virtual machine, with lots of snapshots so I can easily move between configurations to allow me to figure out what's required.

[swillden]

Here's a sequence that works:

1. Install Windows XP, Service Pack 2 (other variations should work, untested)

2. Download and install Python 2.5 from ​http://www.python.org/download/releases/2.5.4/

3. Download and install pywin32 from ​http://sourceforge.net/projects/pywin32/

4. Download and install MinGW from ​http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/MinGW%205.1.4/MinGW-5.1.4.exe/download. The installer will prompt you for what components to install. You don't need any of the optional components, just the base.

5. Configure distutils to use MinGW. To configure distutils to use mingw32, create a file called "distutils.cfg" in C:\Python25\Libs\distutils, and put in that file:

[build]
compiler=mingw32

6. Add the MinGW binary path to your Path variable. To do that, go to the control panel (classic view), double-click "System", click on the "Advanced" tab, click on the "Environment Variables" button, scroll down to the "Path" variable in the "System Variables" list, double-click it, and append ";C:\MinGW\bin" to the path. While you're at it, might as well append ";C:\Python25;C:\Python25\Scripts" also so that you don't have to type full paths to run python or installed python scripts.

7. Download and install the Visual Studio 2008 redistributables from ​http://www.microsoft.com/downloads/details.aspx?familyid=9B2DA534-3E03-4391-8A4D-074B9F2BC1BF&displaylang=en. This is needed for OpenSSL.

8. Download and install OpenSSL v.9.8k Light from ​http://www.slproweb.com/products/Win32OpenSSL.html.

9. Download the latest Tahoe snapshot from ​http://allmydata.org/source/tahoe/snapshots/. Unpack it. I put it in C:\tahoe

10. Open a command prompt and cd to the top of the Tahoe tree (e.g. cd \tahoe).

11. Run "python setup.py build". Wait a bit until it stops working. Note that you need a working network connection because it will download various dependencies.

12. Run "python setup.py install". Wait a bit until it stops working. Again, you may need a working network connection.

13. Run "tahoe create-client --basedir C:\tahoebase". You can pick a different base directory if you want.

14. Run "notepad C:\tahoebase\tahoe.cfg" to edit your config file. Paste in your introducer FURL.

15. Run "tahoe start --basedir C:\taboebase". Your node will start running and connect to the grid! The Windows firewall may ask whether or not to allow "python" to make network connections. Say yes.

Easy peasy! ;-)

Actually, all of the MinGW steps (steps 4-6) could be omitted if win32 eggs were available for all dependencies. I think zfec being out of date is currently the only issue requiring installing a compiler.

[swillden]

And this works with 2.6:

1. Install Windows XP, Service Pack 2 (other variations should work, untested)

2. Download and install Python 2.6.2 from ​http://www.python.org/ftp/python/2.6.2/python-2.6.2.msi

3. Download and install pywin32 from ​http://sourceforge.net/projects/pywin32/files/pywin32/Build%20214/pywin32-214.win32-py2.6.exe/download

4. Download and install MinGW from ​http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/MinGW%205.1.4/MinGW-5.1.4.exe/download. The installer will prompt you for what components to install. You need the base components plus the C++ compiler

5. Configure distutils to use MinGW. To configure distutils to use mingw32, create a file called "distutils.cfg" in C:\Python25\Libs\distutils, and put in that file:

[build]
compiler=mingw32

6. Add the MinGW binary path to your Path variable. To do that, go to the control panel (classic view), double-click "System", click on the "Advanced" tab, click on the "Environment Variables" button, scroll down to the "Path" variable in the "System Variables" list, double-click it, and append ";C:\MinGW\bin" to the path. While you're at it, might as well append ";C:\Python25;C:\Python25\Scripts" also so that you don't have to type full paths to run python or installed python scripts.

7. Download the latest Tahoe snapshot from ​http://allmydata.org/source/tahoe/snapshots/. Unpack it. I put it in C:\tahoe

8. Open a command prompt and cd to the top of the Tahoe tree (e.g. cd \tahoe).

9. Run "python setup.py build". Wait a bit until it stops working. Note that you need a working network connection because it will download various dependencies.

10. Run "python setup.py install". Wait a bit until it stops working. Again, you may need a working network connection.

11. Run "tahoe create-client --basedir C:\tahoebase". You can pick a different base directory if you want.

12. Run "notepad C:\tahoebase\tahoe.cfg" to edit your config file. Paste in your introducer FURL.

13. Run "tahoe start --basedir C:\taboebase". Your node will start running and connect to the grid! The Windows firewall may ask whether or not to allow "python" to make network connections. Say yes.

Actually, all of the MinGW steps (steps 4-6) could be omitted if win32 eggs were available for all dependencies. I think zfec and pycryptopp being out of date are currently the only issues requiring installing a compiler.

[zooko]

Let's see... According to ​http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/ we have zfec-1.4.4-py2.5-win32.egg and zfec-1.4.4-py2.6-win32.egg . Since _auto_deps.py specifies zfec >= 1.1.0, then this ought to satisfy. Is zfec being rebuilt from source when you do this process?

As for pycryptopp, we currently have pycryptopp-0.5.16-py2.5-win32.egg and pycryptopp-0.5.12-py2.6-win32.egg. The one for py2.6 is too old a version of pycryptopp, so I would expect pycryptopp to get built from source if using Python 2.6 but not if using Python 2.5. Is this consistent with your process?

Thanks!

[swillden]

zfec is being rebuilt from source for 2.5. I just started the 2.5 + pywin32 snapshot and re-verified. I believe that's the only reason I need a compiler for 2.5.

It appears that the reason zfec is being rebuilt is that the tarball is 1.4.5 while the egg is 1.4.4. Both satify the requirement, but setup takes the newer one. That seems like a bug to me, and one that may often require the ability to build from source when it isn't really necessary.

pycryptopp is being rebuilt from source for 2.6, but not for 2.5.

The other oddity is that the pyOpenSSL for 2.5 needs OpenSSL DLLs to be installed separately, but pyOpenSSL for 2.6 does not.

[swillden]

Oh, one more thing: Removing the "sdists" URL from the setup.cfg find_list allows the build to complete. The zfec egg is used then.

[zooko]

Added a ticket to setuptools: ​http://bugs.python.org/setuptools/issue83 (I wish setuptools would prefer v1.4.1 in binary egg format over v1.4.5 in sdist format when I'm requiring >= v1.1.0).

[swillden]

Replying to zooko:

Added a ticket to setuptools: ​http://bugs.python.org/setuptools/issue83 (I wish setuptools would prefer v1.4.1 in binary egg format over v1.4.5 in sdist format when I'm requiring >= v1.1.0).

Until your wish is fulfilled, perhaps it would be good to offer a "binary-only" 1.5.0 release that does not have the sdist line in setup.cfg. For systems without development tools installed, removing the sdist line will make the installation more likely to succeed.

Even better (for Windows) is to get a working py2exe installer that "Just Works", of course.

[zooko]

Even better (for Windows) is to get a working py2exe installer that "Just Works", of course.

I hope that these are complementary, not alternative, i.e. I hope that we have a well-tested and documented process of building from source including building dependencies from source (layer 1), as well as a good process of building Tahoe-LAFS from source including installing binary dependencies (layer 2) (this one can be much easier than layer 1 -- no C compiling required!), as well as a prebuilt py2exe or bbfreeze installer for Windows (layer 3).

I don't want the existence of the higher layers to mean we allow the lower layers to rot, for example, if people could easily build Tahoe-LAFS by pulling in binary dependencies, but nobody knew how to build the dependencies from source. This was actually the case at one point in the past -- Tahoe-LAFS required pyOpenSSL but nobody -- literally nobody that I was able to find -- knew how to build pyOpenSSL from source for Windows. We had to rely on downloading a prebuilt pyOpenSSL binary from this one web site where it had, as far as we could tell, been abandoned by whoever originally compiled it.

I'm glad that's not the case anymore!

Anyway, until my wish is fulfilled (​http://bugs.python.org/setuptools/issue83 ), I think a reasonable policy for layer 2 is that we don't upload sdists of dependencies without also uploading accompanying bdists, at least for Windows and probably also Mac. I've already automated most of this for pycryptopp (using Brian's excellent new flappserver tool, plus his excellent old buildbot tool), and I guess I'll proceed to do similar automation for zfec next...

[zooko]

Okay I hooked up allmydata.com's Windows buildslave to upload binary .egg's for Python 2.5 for Win32. Now I guess if you are using Python 2.5 there should be *no* compilation required, so steps 4-6 of Shawn's process in comment:5 should be unnecessary. If we get a Windows buildbot running Python 2.6 working, then this should be true for Python 2.6 as well. (Black Dew has contributed one, but at the moment it is very confused and fails to "download" the setuptools (actually zetuptoolz) .egg from the misc/dependencies/ subdirectory. :-(

​http://allmydata.org/buildbot-zfec/builders/BlackDew%20windows-mingw32-py26

[zooko]

Brian: what else do we need to do besides upload the debs and run the update-apt-repo flapjack?

[swillden]

I'll fire up my 2.5 VM and retest.

[swillden]

Replying to zooko:

Okay I hooked up allmydata.com's Windows buildslave to upload binary .egg's for Python 2.5 for Win32. Now I guess if you are using Python 2.5 there should be *no* compilation required, so steps 4-6 of Shawn's process in comment:5 should be unnecessary.

I've confirmed this. Installation of OpenSSL is still necessary, but no compilation is needed for 2.5.

[warner]

zooko: er, what? Was that comment about uploading debs perhaps meant for a different (non-windows) ticket?

[zooko]

warner: yes it was, sorry about that, and you've already answered it.

[zooko]

This ticket is now renamed from "run through installation on Windows" (thank you Cyberaxe, Ludo', David-Sarah, Black Dew, Michael Walsh, and especially Shawn Willden!) to "document installation on Windows". To close this ticket, collect the wisdom from this ticket and from these mailing list threads and make a wiki page that will help other people install Tahoe-LAFS on Windows:

​http://allmydata.org/pipermail/tahoe-dev/2009-July/002392.html ​http://allmydata.org/pipermail/tahoe-dev/2009-July/002449.html ​http://allmydata.org/pipermail/tahoe-dev/2009-July/002450.html

[zooko]

I think we should announce Tahoe-LAFS v1.5.0 Saturday even if we don't have this ticket fixed and even if #769 (sudo apt-get install allmydata-tahoe) isn't fixed. I highly value those tickets, but this release is just taking way too long and I want to get it over with and move on! We can fix these tickets in the days following the v1.5.0 release.

I'm moving this ticket from "v1.5.0" Milestone to "v1.5.1" Milestone.

[swillden]

I added a description of the Windows install process to InstallDetails.

[zooko]

Thank you Shawn! I'll review your InstallDetails notes sometime and then close this ticket.

[zooko]

This ticket is ready for review -- if someone reads through Shawn's Windows build instructions and tries it out, then they can close this ticket.

[zooko]

Thanks, Shawn and David-Sarah.

​http://allmydata.org/pipermail/tahoe-dev/2009-August/002627.html ​http://allmydata.org/pipermail/tahoe-dev/2009-August/002632.html ​http://allmydata.org/pipermail/tahoe-dev/2009-August/002639.html