[tahoe-dev] problems with the build/install/test process

[Christopher Galvan]

zooko wrote:
> Folks:
>
> Chris Galvan contributed some patches to our buildbot config to make  
> each buildslave test what happens when you do commands like:
>
> python setup.py build_tahoe
> python setup.py trial
> ./bin/tahoe --version
> python setup.py bdist_egg
> mkdir installdir && easy_install -d installdir dist/*.egg
> PATH=$PWD/support/bin:$PATH && PYTHONPATH=$PWD/installdir:$PWD/ 
> installdir/support/lib/python2.5/site-packages && cd installdir &&  
> trial allmydata
>
> It's great to have these tested -- thanks, Chris! -- but currently  
> our setup.py fails to do the right thing in many of these cases.
>
> Here are my notes as I look at our numerous failing builders.
>
> First: cygwin
>
> http://allmydata.org/buildbot/builders/cygwin/builds/1660/steps/ 
> installtest/logs/stdio
>
> This failure is because setup.py doesn't pass "--reactor=poll"  
> whenever starting twisted, including when starting trial.  I guess  
> that the best way to solve this is to add a "reactor" parameter to  
> the setuptools_trial plugin, but I'm not sure the best way to do  
> that, so I think I'll just hack the sys.argv to append "-- 
> reactor=poll" until Chris tells me a better way to do it.  :-)
>
> There, done in the kludgey way in patches [3421] and [3422].
>   

I'm not really sure what would be the best way to implement this.  There 
is a '--reactor=foo' option available with the 'trial' command that the 
setuptools_trial plugin adds, but the problem here is that Brian wants 
to just be able to run 'python setup.py trial' and have it do the right 
thing on any platform, which I agree with.  In order to accomplish this, 
we may need to end up doing something with an extension point being 
offered by the setuptools_trial plugin itself.

>
> Next: edgy, etch,
>
> http://allmydata.org/buildbot/builders/edgy/builds/1874/steps/ 
> installtest/logs/stdio
>
> This is the problem of what to do when a dependency is installed, and  
> Tahoe requires a newer version of that dependency.  Ideally, I would  
> like for the tahoe build process to install the newer version next to  
> the older already-installed version, and arrange to use the newer  
> version at runtime.  Setuptools comes with this functionality, but I  
> don't know how to use that functionality when doing a plugin like  
> "trial" -- I know how to do it only when installing with easy_install  
> or when running "setup.py develop".  It is called "--multi-version",  
> and is the subject of ticket #530 (use setuptools's --multi-version  
> mode).
>   

I haven't looked much into ticket #530 and will need to do so in order 
to have a better answer here :)

>
> Next: hardy
>
> http://allmydata.org/buildbot/builders/hardy/builds/480/steps/test- 
> egg/logs/stdio
>
> Hm, this points to a deeper problem: the buildbot's attempt to run  
> "trial allmydata" to test the current source code from its newly  
> installed location might accidentally run the unit tests of a  
> different version of allmydata installed in the system.  This could  
> be related to #145 ("make test" tests the installed version of  
> allmydata, not the local sandbox version of allmydata).
>   

I agree, this one may be a little bit harder to track down :/  The 
traceback given by buildbot is not very helpful.

>
> Next:
>
> Next: Mac OS X
>
> http://allmydata.org/buildbot/builders/Mac%20OS-X/builds/806/steps/ 
> mac_exe/logs/stdio
>
> Hm.  The building of Macintosh packages is currently unsupported, but  
> the changes we've made shouldn't have broken the tests that were  
> previously passing.
>
> By the way, does anybody want to take over the code to automatically  
> build Macintosh packages of Tahoe?  There are many people who have  
> expressed desire for such packages.
>   

If no one else wants to, I have access to both OS X 10.4 and 10.5 intel 
machines, and an OS X 10.4 ppc machine that I can probably setup build 
slaves on.

-- Chris Galvan

>
> Regards,
>
> Zooko
>
> patches mentioned in this mail:
>
> http://allmydata.org/trac/tahoe/changeset/3421
> http://allmydata.org/trac/tahoe/changeset/3422
>
> tickets mentioned in this mail:
>
> http://allmydata.org/trac/tahoe/ticket/530 # use setuptools's --multi- 
> version mode
> http://allmydata.org/trac/tahoe/ticket/145 # "make test" tests the  
> installed version of allmydata, not the local sandbox version of  
> allmydata
>
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at allmydata.org
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>
>
>