clean up after yourself when told to "make clean"

[zooko]

make "./setup.py clean --all" really clean up properly

[zooko]

I'd like to add a test to be executed by the buildslave that does "make ; make clean" and then asserts that "darcs whatsnew -s -l" shows no new files.

[zooko]

#6 merged into here

[warner]

I'll work on this. I noticed that 'make; darcs whatsnew -s -l' shows no new files: i.e., our .darcs-boringfile is doing a great job of ignoring generated files, which makes darcs less-than-useful to detect generated files :-).

I think that I'll build two tests. One will verify that .darcs-boringfile is doing its job, and will pass if 'make; darcs whatsnew -s -l' says 'No changes!'. The second will verify that we clean up after ourselves, and will be something like:

find . -ls |sort >allfiles.tmp.old
make
make clean
find . -ls |grep -v allfiles.tmp |sort >allfiles.tmp.new
diff allfiles.tmp.old allfiles.tmp.new

and see if there are any differences.

[warner]

I've added these tests, in 319516a5a212cdf3, 13e2af80339eceeb, and 8b9d7d6306137b86. The test-darcs-boringfile test should be passing now, but the test-clean check is failing. The problems fall into the following categories:

• src/allmydata/version.py
• build-in-place .pyc files for foolscap
• src/zfec/ez_setup.pyc
• src/zfec/zfec.egg-info

[warner]

I spent a few minutes trying to figure out why foolscap has .pyc files in place, with no luck. 'make build-foolscap' is sufficient to cause src/foolscap/foolscap/pb.pyc to be created. We're not doing a 'setup.py build -i' anywhere, and I can't find anywhere that might be adding src/foolscap to PYTHONPATH and then importing foolscap modules. I see two copies of pb.pyc created: the expected one in instdir/lib/foolscap/pb.pyc and the unwanted one in src/foolscap/foolscap/pb.pyc. The instdir one is easy to clean up, but the src/foolscap one isn't (I don't want to add a 'find *.pyc -exec rm' statement to the clean target, since I'd much prefer to put all the generated files in one place rather than go traipsing through source directories looking for them).

src/allmydata/version.py is generated by all build targets, and should probably be deleted by the clean target. I have no idea what zfec and eggs are up to.

[zooko]

Note to self: I really want the buildbot status to be all green when we release v0.5.

This isn't a "blocker" -- it isn't that important -- but I'm putting it in the v0.5 milestone to remind myself when I look at this link tomorrow:

​http://allmydata.org/trac/tahoe/query?status=new&status=assigned&status=reopened&groupdesc=1&group=priority&component=code&milestone=0.5.0&order=priority

[zooko]

I have asked for help from the setuptools experts at the distutils-sig mailing list.

My current hypothesis is that setuptools ought to be amended to *not* product the .egg-info directory when the --single-version-externally-managed flag is present.

[zooko]

bumping this to v0.6

[warner]

With 8088044bf3a7dc5d, our 'make clean' target restores the tree to its original state, and the buildbot 'clean' builder is now green. This uses clean-zfec and clean-setuptools targets that explicitly delete all of the random .pyc and egg.info and setuptools.egg files that are created during their building, which may not be as satisfactory a solution as changing the setup.py files to do this in the 'setup.py clean' target.. I'll leave that decision up to zooko.

[warner]

are we happy with how this works now?

[zooko]

Yep! It is all good. Now, for the record, we no longer have clean-zfec and clean-setuptools targets, but we do explicitly delete random .pyc and such, as per this excerpt from the Makefile:

clean:
	rm -rf build _trial_temp _test_memory .checked-deps .built
	rm -f debian
	rm -f `find src/allmydata -name '*.so' -or -name '*.pyc'`
	rm -rf tahoe_deps.egg-info allmydata_tahoe.egg-info
	rm -rf support dist
	rm -rf setuptools*.egg *.pyc