buildbot is unable to delete directories under _trial_temp* that have non-ASCII names under Windows

[davidsarah]

The drop-upload tests added in 32a7717205ed824a create a subdirectory under _trial_temp that has a non-ASCII name (locāl_dir). On Windows, this causes an error subsequent builds when buildbot tries to delete the "workdir" for the build, which contains _trial_temp. This seems to happen because it ASCIIfies the name to "local_dir" when trying to change its permissions to ensure it is deleteable.

An ugly workaround would be for the tests to delete any non-ASCII directories they create. This should really be fixed in buildbot, though.

The traceback is:

starting darcs operation

remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last):
  File "c:\python27\lib\site-packages\buildbot-0.7.12-py2.7.egg\buildbot\slave\commands.py", line 1425, in start
    self.maybeClobber(d)
  File "c:\python27\lib\site-packages\buildbot-0.7.12-py2.7.egg\buildbot\slave\commands.py", line 1444, in maybeClobber
    d.addCallback(self.doClobber, self.workdir)
  File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 297, in addCallback
    callbackKeywords=kw)
  File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 286, in addCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 542, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "c:\python27\lib\site-packages\buildbot-0.7.12-py2.7.egg\buildbot\slave\commands.py", line 1600, in doClobber
    rmdirRecursive(d)
  File "c:\python27\lib\site-packages\buildbot-0.7.12-py2.7.egg\buildbot\slave\commands.py", line 127, in rmdirRecursive
    rmdirRecursive(full_name)
  File "c:\python27\lib\site-packages\buildbot-0.7.12-py2.7.egg\buildbot\slave\commands.py", line 127, in rmdirRecursive
    rmdirRecursive(full_name)
  File "c:\python27\lib\site-packages\buildbot-0.7.12-py2.7.egg\buildbot\slave\commands.py", line 124, in rmdirRecursive
    os.chmod(full_name, 0600)
exceptions.WindowsError: [Error 2] Le fichier spécifié est introuvable: 'C:\\buildbot_tahoe\\FreeStorm_WinXP-x86_py2.6\\build\\_trial_temp_old228388\\drop_upload.MockTest.test_drop_upload\\local_dir'
]

[davidsarah]

70f03bd6158168ed should work around this.

[davidsarah]

Bug filed against buildbot: ​http://trac.buildbot.net/ticket/2073

[david-sarah@…]

In db22fdc20dc93a3e:

Factor out methods dealing with non-ASCII directories and filenames from test_drop_upload.py into common_util.py. refs #1429, #1472

[david-sarah@…]

In 2deba3319cbf02c8:

(The changeset message doesn't reference this ticket)

[david-sarah@…]

In 0dc56daaf9540910:

test/common_util.py: correct fix to mkdir_nonascii. refs #1472

[david-sarah@…]

In [5163/ticket393-MDMF-2]:

Factor out methods dealing with non-ASCII directories and filenames from test_drop_upload.py into common_util.py. refs #1429, #1472

[david-sarah@…]

In [5165/ticket393-MDMF-2]:

test/common_util.py: fix a typo. refs #1472

[david-sarah@…]

In [5166/ticket393-MDMF-2]:

test/common_util.py: correct fix to mkdir_nonascii. refs #1472

[davidsarah]

This is strictly speaking invalid now for Tahoe-LAFS, since the actual bug is in buildbot and we've worked around the immediate problem. However, when buildbot is fixed we'll need to decide whether to remove the workaround.

In the meantime, other tests that create non-ASCII directory names should mix in NonASCIIPathMixin from src/allmydata/test/common_util.py​ and use self.mkdir_nonascii.

[zooko]

FWIW, there is a class in fileutil.py that does something similar: src/allmydata/util/fileutil.py@4957#L90. There is also a more featureful version of that class in pyutil: ​pyutil/fileutil.py.

[davidsarah]

Replying to zooko:

FWIW, there is a class in fileutil.py that does something similar: src/allmydata/util/fileutil.py@4957#L90.

NamedTemporaryDirectory deletes the directory tree when the NamedTemporaryDirectory object is __del__'d, which is the wrong time. We need to delete it when the test has finished (which, because of Deferreds, will be some time after the test method returns). There's no simpler way to do this than using addCleanup, as the current code does.

[zooko]

Replying to davidsarah:

Replying to zooko:

FWIW, there is a class in fileutil.py that does something similar: src/allmydata/util/fileutil.py@4957#L90.

NamedTemporaryDirectory deletes the directory tree when the NamedTemporaryDirectory object is __del__'d, which is the wrong time.

For future reference, you can also invoke the NamedTemporaryDirectory object's shutdown method at a specific time if desired.

[exarkun]

Buildbot has been decommissioned.