automatically generate binary .egg's of pycryptopp for all of our Supported Platforms

[zooko]

zeromus's #1093 (win32 build hell) was at least partly if not mostly caused by the following problem:

1. zeromus followed the instructions on ​quickstart.html to use Python 2.5
2. the build script saw from _auto_deps.py that Tahoe-LAFS requires pycryptopp, so it starting looking for where to get pycryptopp to satisfy this dependency
3. it looked in the SUMO tarball that zeromus had installed, which has the same contents as ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-deps-v16/ , and saw therein "pycryptopp-0.5.19.tar.bz2". It also looked on the Python Package Index: ​http://pypi.python.org/pypi/pycryptopp and saw that pycryptopp v0.5.19 is the current version of pycryptopp.
4. it then looked for binary eggs of pycryptopp v0.5.19 to install. From _auto_dep.py's find_links it could see that the following directory was a valid place to get dependencies: ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-eggs/ . At the time, that directory contained the following packages of pycryptopp:

   	pycryptopp-0.5.17_r683-py2.5-win32.egg	10-Jan-2010 21:31	1.1M	 
   	pycryptopp-0.5.17_r683-py2.6-linux-x86_64.egg	10-Jan-2010 21:35	3.0M	 
   	pycryptopp-0.5.17_r683-py2.5-linux-armv5tel.egg	10-Jan-2010 21:47	554K	 
   	pycryptopp-0.5.17-py2.5-macosx-10.5-i386.egg	06-Feb-2010 14:11	1.7M	 
   	pycryptopp-0.5.17_r683-py2.5-macosx-10.3-i386.egg	07-Feb-2010 10:53	2.5M	 
   	pycryptopp-0.5.17_r683-py2.5-macosx-10.4-i386.egg	07-Feb-2010 10:53	2.5M	 
   	pycryptopp-0.5.17_r683-py2.5-macosx-10.5-i386.egg	07-Feb-2010 10:53	2.5M	 
   	pycryptopp-0.5.17_r683-py2.5-macosx-10.6-i386.egg	07-Feb-2010 10:53	2.5M	 
   	pycryptopp-0.5.17_r683-py2.6-macosx-10.6-universal.egg	03-Mar-2010 22:50	2.6M	 
   	pycryptopp-0.5.17.360334-py2.6-win-amd64.egg	01-May-2010 22:14	1.0M	 
   	pycryptopp-0.5.17_r683-py2.6-win32.egg	06-May-2010 14:11	1.1M	 
   	pycryptopp-0.5.17_r685-py2.6-win32.egg	12-May-2010 05:38	1.1M	 
   	pycryptopp-0.5.17_r686-py2.6-win32.egg	15-May-2010 18:40	1.1M	 
   	pycryptopp-0.5.17_r687-py2.6-win32.egg	15-May-2010 18:48	1.1M	 
   	pycryptopp-0.5.17_r688-py2.6-win32.egg	15-May-2010 19:40	1.1M	 
   	pycryptopp-0.5.17_r689-py2.6-win32.egg	15-May-2010 19:43	1.1M	 
   	pycryptopp-0.5.17_r693-py2.6-win32.egg	03-Jun-2010 23:17	1.1M	 
   	pycryptopp-0.5.18_r701-py2.6-win32.egg	03-Jun-2010 23:53	1.1M	 
   	pycryptopp-0.5.19-py2.6-win32.egg	04-Jun-2010 00:06	1.1M	 
   	pycryptopp-0.5.19_r704-py2.6-win32.egg	06-Jun-2010 23:58	1.1M	 
   	pycryptopp-0.5.19_r704-py2.6-win-amd64.egg	08-Jun-2010 07:37	602K	 
   

Now it can't use any binary eggs that aren't for win32 or that aren't for Python 2.5, so that leaves:

	pycryptopp-0.5.17_r683-py2.5-win32.egg	10-Jan-2010 21:31	1.1M	 

But, it will prefer to use a newer release from source tarball over using an older release from a binary egg! Since that py2.5-win32.egg there is pycryptopp v0.5.17, and since it can get pycryptopp v0.5.19 in source tarball form, it will go for the latter.

Therefore, it went ahead and used the pycryptopp v0.5.19 source tarball from the SUMO tarball and tried to compile it.

It then failed to compile it with this error:

error: Setup script exited with error: Don't know how to compile cryptopp\x64dll.asm

I believe that failure is due to ​http://tahoe-lafs.org/trac/pycryptopp/ticket/37.

However, that failure to build is not the subject of this ticket. The subject of this ticket is: why didn't the build system find a binary egg of pycryptopp for the current platform and use that?

Ok, why does ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-eggs have new binary eggs of pycryptopp v0.5.19 for py2.6-win32 and for py2.6-win-amd64.egg but not for py2.5-win32? The answer is easy to see by looking at ​the pycryptopp buildbot page. There is a buildslave for py2.6-win32 (operated by FreeStorm) and a buildslave for py2.6-win-amd64 (operated by Dcoder/sneves). There is no buildslave for py2.5-win32. That means that py2.5-win32 is not a Supported Platform!

So to close this ticket, either change the docs to specify that you should use Python 2.6 and if you use any other version of Python then you are on your own, or else get someone to volunteer a py2.5-win32 buildslave to generate pycryptopp binary eggs.

[davidsarah]

Replying to zooko:

zeromus's #1093 (win32 build hell) was at least partly if not mostly caused by the following problem:

1. zeromus followed the instructions on ​quickstart.html to use Python 2.5

[...]

Ok, why does ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-eggs have new binary eggs of pycryptopp v0.5.19 for py2.6-win32 and for py2.6-win-amd64.egg but not for py2.5-win32? The answer is easy to see by looking at ​the pycryptopp buildbot page. There is a buildslave for py2.6-win32 (operated by FreeStorm) and a buildslave for py2.6-win-amd64 (operated by Dcoder/sneves). There is no buildslave for py2.5-win32. That means that py2.5-win32 is not a Supported Platform!

So to close this ticket, either change the docs to specify that you should use Python 2.6 and if you use any other version of Python then you are on your own, or else get someone to volunteer a py2.5-win32 buildslave to generate pycryptopp binary eggs.

If I understand correctly, the original reason for recommending Python 2.5 on Windows, was that mingw cannot compile extension modules for Python 2.6 on Windows; that requires a Microsoft compiler. Since we have py2.6-win32 and py2.6-win32-amd64 buildslaves with Microsoft compilers, this is presumably less of an issue now (at least for users who do not want to compile everything from scratch, but those are not the audience quickstart.html is targeted at).

[davidsarah]

5ade4f82e0ee38cf updated quickstart.html to recommend use of Python 2.6 on Windows. Should we consider this fixed?

[zooko]

Okay we have automatic generation of pycryptopp bdist_eggs on our buildbot. You can see its logs here:

​http://tahoe-lafs.org/buildbot-pycryptopp/waterfall

and the resulting .egg's are hosted here: ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-eggs/

We can close this ticket, about automating this process, but we're going to open a new ticket about visualizing which platforms and python versions have which versions of which dependencies available (i.e. by writing a script that generates a .rst table)...

That visualization would then inform us about which versions of Python we can recommend on which platforms...