Ticket #1076: nfc-normalization-2.dpatch

Wed Jun 16 04:14:50 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * Provisional patch to NFC-normalize filenames going in and out of Tahoe directories.

Wed Jun 16 05:20:12 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * stringutils.py: Add encoding argument to quote_output. Also work around a bug in locale.getpreferredencoding on older Pythons.

Thu Jun 17 02:55:37 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * stringutils.py: don't NFC-normalize the output of listdir_unicode.

Thu Jun 17 04:39:01 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * CLI: allow Unicode patterns in exclude option to 'tahoe backup'.

Thu Jun 17 04:40:25 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * test_dirnode.py: partial tests for normalization changes.

Thu Jun 17 04:44:09 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * test_stringutils.py: take account of the output of listdir_unicode no longer being normalized. Also use Unicode escapes, not UTF-8.

Thu Jun 17 04:44:40 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * stringutils.py: remove unused import.

Thu Jun 17 05:14:11 GMT Daylight Time 2010  david-sarah@jacaranda.org
  * dirnode.py: comments about normalization changes.

New patches:

[Provisional patch to NFC-normalize filenames going in and out of Tahoe directories.
david-sarah@jacaranda.org**20100616031450
 Ignore-this: ed08c9d8df37ef0b7cca42bb562c996b
] {
hunk ./src/allmydata/dirnode.py 2
 
-import time, math
+import time, math, unicodedata
 
 from zope.interface import implements
 from twisted.internet import defer
hunk ./src/allmydata/dirnode.py 19
      DeepCheckAndRepairResults
 from allmydata.monitor import Monitor
 from allmydata.util import hashutil, mathutil, base32, log
+from allmydata.util.stringutils import quote_output
 from allmydata.util.assertutil import precondition
 from allmydata.util.netstring import netstring, split_netstring
 from allmydata.util.consumer import download_to_data
hunk ./src/allmydata/dirnode.py 79
 
     return metadata
 
+def normalize(namex):
+    return unicodedata.normalize('NFC', namex)
 
 # TODO: {Deleter,MetadataSetter,Adder}.modify all start by unpacking the
 # contents and end by repacking them. It might be better to apply them to
hunk ./src/allmydata/dirnode.py 87
 # the unpacked contents.
 
 class Deleter:
-    def __init__(self, node, name, must_exist=True, must_be_directory=False, must_be_file=False):
+    def __init__(self, node, namex, must_exist=True, must_be_directory=False, must_be_file=False):
         self.node = node
hunk ./src/allmydata/dirnode.py 89
-        self.name = name
+        self.name = normalize(namex)
         self.must_exist = must_exist
         self.must_be_directory = must_be_directory
         self.must_be_file = must_be_file
hunk ./src/allmydata/dirnode.py 115
 
 
 class MetadataSetter:
-    def __init__(self, node, name, metadata, create_readonly_node=None):
+    def __init__(self, node, namex, metadata, create_readonly_node=None):
         self.node = node
hunk ./src/allmydata/dirnode.py 117
-        self.name = name
+        self.name = normalize(namex)
         self.metadata = metadata
         self.create_readonly_node = create_readonly_node
 
hunk ./src/allmydata/dirnode.py 145
         if entries is None:
             entries = {}
         precondition(isinstance(entries, dict), entries)
+        # keys of 'entries' may not be normalized.
         self.entries = entries
         self.overwrite = overwrite
         self.create_readonly_node = create_readonly_node
hunk ./src/allmydata/dirnode.py 150
 
-    def set_node(self, name, node, metadata):
-        precondition(isinstance(name, unicode), name)
+    def set_node(self, namex, node, metadata):
         precondition(IFilesystemNode.providedBy(node), node)
hunk ./src/allmydata/dirnode.py 152
-        self.entries[name] = (node, metadata)
+        self.entries[namex] = (node, metadata)
 
     def modify(self, old_contents, servermap, first_time):
         children = self.node._unpack_contents(old_contents)
hunk ./src/allmydata/dirnode.py 157
         now = time.time()
-        for (name, (child, new_metadata)) in self.entries.iteritems():
-            precondition(isinstance(name, unicode), name)
+        for (namex, (child, new_metadata)) in self.entries.iteritems():
+            name = normalize(namex)
             precondition(IFilesystemNode.providedBy(child), child)
 
             # Strictly speaking this is redundant because we would raise the
hunk ./src/allmydata/dirnode.py 168
             metadata = None
             if name in children:
                 if not self.overwrite:
-                    raise ExistingChildError("child '%s' already exists" % name)
+                    raise ExistingChildError("child %s already exists" % quote_output(name, encoding='utf-8'))
 
                 if self.overwrite == "only-files" and IDirectoryNode.providedBy(children[name][0]):
hunk ./src/allmydata/dirnode.py 171
-                    raise ExistingChildError("child '%s' already exists" % name)
+                    raise ExistingChildError("child %s already exists" % quote_output(name, encoding='utf-8'))
                 metadata = children[name][1].copy()
 
             metadata = update_metadata(metadata, new_metadata, now)
hunk ./src/allmydata/dirnode.py 182
         new_contents = self.node._pack_contents(children)
         return new_contents
 
+
 def _encrypt_rw_uri(filenode, rw_uri):
     assert isinstance(rw_uri, str)
     writekey = filenode.get_writekey()
hunk ./src/allmydata/dirnode.py 219
         (child, metadata) = children[name]
         child.raise_error()
         if deep_immutable and not child.is_allowed_in_immutable_directory():
-            raise MustBeDeepImmutableError("child '%s' is not allowed in an immutable directory" % (name,), name)
+            raise MustBeDeepImmutableError("child %s is not allowed in an immutable directory" %
+                                           quote_output(name, encoding='utf-8'), name)
         if has_aux:
             entry = children.get_aux(name)
         if not entry:
hunk ./src/allmydata/dirnode.py 292
         return plaintext
 
     def _create_and_validate_node(self, rw_uri, ro_uri, name):
+        # name is just for error reporting
         node = self._nodemaker.create_from_cap(rw_uri, ro_uri,
                                                deep_immutable=not self.is_mutable(),
                                                name=name)
hunk ./src/allmydata/dirnode.py 300
         return node
 
     def _create_readonly_node(self, node, name):
+        # name is just for error reporting
         if not node.is_unknown() and node.is_readonly():
             return node
         return self._create_and_validate_node(None, node.get_readonly_uri(), name=name)
hunk ./src/allmydata/dirnode.py 309
         # the directory is serialized as a list of netstrings, one per child.
         # Each child is serialized as a list of four netstrings: (name, ro_uri,
         # rwcapdata, metadata), in which the name, ro_uri, metadata are in
-        # cleartext. The 'name' is UTF-8 encoded. The rwcapdata is formatted as:
+        # cleartext. The 'name' is UTF-8 encoded, and should be normalized to NFC.
+        # The rwcapdata is formatted as:
         # pack("16ss32s", iv, AES(H(writekey+iv), plaintext_rw_uri), mac)
         assert isinstance(data, str), (repr(data), type(data))
         # an empty directory is serialized as an empty string
hunk ./src/allmydata/dirnode.py 323
         while position < len(data):
             entries, position = split_netstring(data, 1, position)
             entry = entries[0]
-            (name_utf8, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4)
+            (namex_utf8, ro_uri, rwcapdata, metadata_s), subpos = split_netstring(entry, 4)
             if not mutable and len(rwcapdata) > 0:
                 raise ValueError("the rwcapdata field of a dirnode in an immutable directory was not empty")
hunk ./src/allmydata/dirnode.py 326
-            name = name_utf8.decode("utf-8")
+
+            # A name containing characters that are unassigned in one version of Unicode might
+            # not be normalized wrt a later version. Therefore we normalize names going both in
+            # and out of directories.
+            name = normalize(namex_utf8.decode("utf-8"))
+
             rw_uri = ""
             if writeable:
                 rw_uri = self._decrypt_rwcapdata(rwcapdata)
hunk ./src/allmydata/dirnode.py 354
                     children[name] = (child, metadata)
                     children.set_with_aux(name, (child, metadata), auxilliary=entry)
                 else:
-                    log.msg(format="mutable cap for child '%(name)s' unpacked from an immutable directory",
-                                   name=name_utf8,
+                    log.msg(format="mutable cap for child %(name)s unpacked from an immutable directory",
+                                   name=quote_output(name, encoding='utf-8'),
                                    facility="tahoe.webish", level=log.UNUSUAL)
             except CapConstraintError, e:
hunk ./src/allmydata/dirnode.py 358
-                log.msg(format="unmet constraint on cap for child '%(name)s' unpacked from a directory:\n"
-                               "%(message)s", message=e.args[0], name=name_utf8,
+                log.msg(format="unmet constraint on cap for child %(name)s unpacked from a directory:\n"
+                               "%(message)s", message=e.args[0], name=quote_output(name, encoding='utf-8'),
                                facility="tahoe.webish", level=log.UNUSUAL)
 
         return children
hunk ./src/allmydata/dirnode.py 422
         name to a tuple of (IFilesystemNode, metadata)."""
         return self._read()
 
-    def has_child(self, name):
+    def has_child(self, namex):
         """I return a Deferred that fires with a boolean, True if there
         exists a child of the given name, False if not."""
hunk ./src/allmydata/dirnode.py 425
-        assert isinstance(name, unicode)
+        name = normalize(namex)
         d = self._read()
         d.addCallback(lambda children: children.has_key(name))
         return d
hunk ./src/allmydata/dirnode.py 442
             raise NoSuchChildError(name)
         return child
 
-    def get(self, name):
+    def get(self, namex):
         """I return a Deferred that fires with the named child node,
         which is an IFilesystemNode."""
hunk ./src/allmydata/dirnode.py 445
-        assert isinstance(name, unicode)
+        name = normalize(namex)
         d = self._read()
         d.addCallback(self._get, name)
         return d
hunk ./src/allmydata/dirnode.py 450
 
-    def get_child_and_metadata(self, name):
+    def get_child_and_metadata(self, namex):
         """I return a Deferred that fires with the (node, metadata) pair for
         the named child. The node is an IFilesystemNode, and the metadata
         is a dictionary."""
hunk ./src/allmydata/dirnode.py 454
-        assert isinstance(name, unicode)
+        name = normalize(namex)
         d = self._read()
         d.addCallback(self._get_with_metadata, name)
         return d
hunk ./src/allmydata/dirnode.py 459
 
-    def get_metadata_for(self, name):
-        assert isinstance(name, unicode)
+    def get_metadata_for(self, namex):
+        name = normalize(namex)
         d = self._read()
         d.addCallback(lambda children: children[name][1])
         return d
hunk ./src/allmydata/dirnode.py 465
 
-    def set_metadata_for(self, name, metadata):
-        assert isinstance(name, unicode)
+    def set_metadata_for(self, namex, metadata):
+        name = normalize(namex)
         if self.is_readonly():
             return defer.fail(NotWriteableError())
         assert isinstance(metadata, dict)
hunk ./src/allmydata/dirnode.py 476
         d.addCallback(lambda res: self)
         return d
 
-    def get_child_at_path(self, path):
+    def get_child_at_path(self, pathx):
         """Transform a child path into an IFilesystemNode.
 
         I perform a recursive series of 'get' operations to find the named
hunk ./src/allmydata/dirnode.py 486
         The path can be either a single string (slash-separated) or a list of
         path-name elements.
         """
-        d = self.get_child_and_metadata_at_path(path)
+        d = self.get_child_and_metadata_at_path(pathx)
         d.addCallback(lambda (node, metadata): node)
         return d
 
hunk ./src/allmydata/dirnode.py 490
-    def get_child_and_metadata_at_path(self, path):
+    def get_child_and_metadata_at_path(self, pathx):
         """Transform a child path into an IFilesystemNode and
         a metadata dictionary from the last edge that was traversed.
         """
hunk ./src/allmydata/dirnode.py 495
 
-        if not path:
+        if not pathx:
             return defer.succeed((self, {}))
hunk ./src/allmydata/dirnode.py 497
-        if isinstance(path, (list, tuple)):
+        if isinstance(pathx, (list, tuple)):
             pass
         else:
hunk ./src/allmydata/dirnode.py 500
-            path = path.split("/")
-        for p in path:
-            assert isinstance(p, unicode)
-        childname = path[0]
-        remaining_path = path[1:]
-        if remaining_path:
-            d = self.get(childname)
+            pathx = pathx.split("/")
+        for p in pathx:
+            assert isinstance(p, unicode), p
+        childnamex = pathx[0]
+        remaining_pathx = pathx[1:]
+        if remaining_pathx:
+            d = self.get(childnamex)
             d.addCallback(lambda node:
hunk ./src/allmydata/dirnode.py 508
-                          node.get_child_and_metadata_at_path(remaining_path))
+                          node.get_child_and_metadata_at_path(remaining_pathx))
             return d
hunk ./src/allmydata/dirnode.py 510
-        d = self.get_child_and_metadata(childname)
+        d = self.get_child_and_metadata(childnamex)
         return d
 
hunk ./src/allmydata/dirnode.py 513
-    def set_uri(self, name, writecap, readcap, metadata=None, overwrite=True):
-        precondition(isinstance(name, unicode), name)
+    def set_uri(self, namex, writecap, readcap, metadata=None, overwrite=True):
         precondition(isinstance(writecap, (str,type(None))), writecap)
         precondition(isinstance(readcap, (str,type(None))), readcap)
 
hunk ./src/allmydata/dirnode.py 519
         # We now allow packing unknown nodes, provided they are valid
         # for this type of directory.
-        child_node = self._create_and_validate_node(writecap, readcap, name)
-        d = self.set_node(name, child_node, metadata, overwrite)
+        child_node = self._create_and_validate_node(writecap, readcap, namex)
+        d = self.set_node(namex, child_node, metadata, overwrite)
         d.addCallback(lambda res: child_node)
         return d
 
hunk ./src/allmydata/dirnode.py 528
         # this takes URIs
         a = Adder(self, overwrite=overwrite,
                   create_readonly_node=self._create_readonly_node)
-        for (name, e) in entries.iteritems():
-            assert isinstance(name, unicode)
+        for (namex, e) in entries.iteritems():
+            assert isinstance(namex, unicode), namex
             if len(e) == 2:
                 writecap, readcap = e
                 metadata = None
hunk ./src/allmydata/dirnode.py 541
             
             # We now allow packing unknown nodes, provided they are valid
             # for this type of directory.
-            child_node = self._create_and_validate_node(writecap, readcap, name)
-            a.set_node(name, child_node, metadata)
+            child_node = self._create_and_validate_node(writecap, readcap, namex)
+            a.set_node(namex, child_node, metadata)
         d = self._node.modify(a.modify)
         d.addCallback(lambda ign: self)
         return d
hunk ./src/allmydata/dirnode.py 547
 
-    def set_node(self, name, child, metadata=None, overwrite=True):
+    def set_node(self, namex, child, metadata=None, overwrite=True):
         """I add a child at the specific name. I return a Deferred that fires
         when the operation finishes. This Deferred will fire with the child
         node that was just added. I will replace any existing child of the
hunk ./src/allmydata/dirnode.py 560
 
         if self.is_readonly():
             return defer.fail(NotWriteableError())
-        assert isinstance(name, unicode)
         assert IFilesystemNode.providedBy(child), child
         a = Adder(self, overwrite=overwrite,
                   create_readonly_node=self._create_readonly_node)
hunk ./src/allmydata/dirnode.py 563
-        a.set_node(name, child, metadata)
+        a.set_node(namex, child, metadata)
         d = self._node.modify(a.modify)
         d.addCallback(lambda res: child)
         return d
hunk ./src/allmydata/dirnode.py 579
         return d
 
 
-    def add_file(self, name, uploadable, metadata=None, overwrite=True):
+    def add_file(self, namex, uploadable, metadata=None, overwrite=True):
         """I upload a file (using the given IUploadable), then attach the
         resulting FileNode to the directory at the given name. I return a
         Deferred that fires (with the IFileNode of the uploaded file) when
hunk ./src/allmydata/dirnode.py 584
         the operation completes."""
-        assert isinstance(name, unicode)
+        name = normalize(namex)
         if self.is_readonly():
             return defer.fail(NotWriteableError())
         d = self._uploader.upload(uploadable)
hunk ./src/allmydata/dirnode.py 594
                       self.set_node(name, node, metadata, overwrite))
         return d
 
-    def delete(self, name, must_exist=True, must_be_directory=False, must_be_file=False):
+    def delete(self, namex, must_exist=True, must_be_directory=False, must_be_file=False):
         """I remove the child at the specific name. I return a Deferred that
         fires (with the node just removed) when the operation finishes."""
hunk ./src/allmydata/dirnode.py 597
-        assert isinstance(name, unicode)
         if self.is_readonly():
             return defer.fail(NotWriteableError())
hunk ./src/allmydata/dirnode.py 599
-        deleter = Deleter(self, name, must_exist=must_exist,
+        deleter = Deleter(self, namex, must_exist=must_exist,
                           must_be_directory=must_be_directory, must_be_file=must_be_file)
         d = self._node.modify(deleter.modify)
         d.addCallback(lambda res: deleter.old_child)
hunk ./src/allmydata/dirnode.py 605
         return d
 
-    def create_subdirectory(self, name, initial_children={}, overwrite=True,
+    def create_subdirectory(self, namex, initial_children={}, overwrite=True,
                             mutable=True, metadata=None):
hunk ./src/allmydata/dirnode.py 607
-        assert isinstance(name, unicode)
+        name = normalize(namex)
         if self.is_readonly():
             return defer.fail(NotWriteableError())
         if mutable:
hunk ./src/allmydata/dirnode.py 624
         d.addCallback(_created)
         return d
 
-    def move_child_to(self, current_child_name, new_parent,
-                      new_child_name=None, overwrite=True):
+    def move_child_to(self, current_child_namex, new_parent,
+                      new_child_namex=None, overwrite=True):
         """I take one of my children and move them to a new parent. The child
         is referenced by name. On the new parent, the child will live under
         'new_child_name', which defaults to 'current_child_name'. I return a
hunk ./src/allmydata/dirnode.py 630
         Deferred that fires when the operation finishes."""
-        assert isinstance(current_child_name, unicode)
+
         if self.is_readonly() or new_parent.is_readonly():
             return defer.fail(NotWriteableError())
hunk ./src/allmydata/dirnode.py 633
-        if new_child_name is None:
-            new_child_name = current_child_name
-        assert isinstance(new_child_name, unicode)
+
+        current_child_name = normalize(current_child_namex)
+        if new_child_namex is None:
+            new_child_namex = current_child_name
         d = self.get(current_child_name)
         def sn(child):
hunk ./src/allmydata/dirnode.py 639
-            return new_parent.set_node(new_child_name, child,
+            return new_parent.set_node(new_child_namex, child,
                                        overwrite=overwrite)
         d.addCallback(sn)
         d.addCallback(lambda child: self.delete(current_child_name))
}
[stringutils.py: Add encoding argument to quote_output. Also work around a bug in locale.getpreferredencoding on older Pythons.
david-sarah@jacaranda.org**20100616042012
 Ignore-this: 48174c37ad95205997e4d3cdd81f1e28
] {
hunk ./src/allmydata/util/stringutils.py 13
 from allmydata.util.assertutil import precondition
 from twisted.python import usage
 import locale
+from allmydata.util import log
 
 
 def _canonical_encoding(encoding):
hunk ./src/allmydata/util/stringutils.py 18
     if encoding is None:
+        log.msg("Warning: falling back to UTF-8 encoding.", level=log.WEIRD)
         encoding = 'utf-8'
     encoding = encoding.lower()
     if encoding == "cp65001":
hunk ./src/allmydata/util/stringutils.py 23
         encoding = 'utf-8'
-    elif encoding == "us-ascii" or encoding == "646":
+    elif encoding == "us-ascii" or encoding == "646" or encoding == "ansi_x3.4-1968":
         encoding = 'ascii'
 
     # sometimes Python returns an encoding name that it doesn't support for conversion
hunk ./src/allmydata/util/stringutils.py 44
     global filesystem_encoding, output_encoding, argv_encoding, is_unicode_platform
 
     filesystem_encoding = _canonical_encoding(sys.getfilesystemencoding())
-    output_encoding = _canonical_encoding(sys.stdout.encoding or locale.getpreferredencoding())
+
+    outenc = sys.stdout.encoding
+    if outenc is None:
+        try:
+            outenc = locale.getpreferredencoding()
+        except Exception:
+            pass  # work around <http://bugs.python.org/issue1443504>
+    output_encoding = _canonical_encoding(outenc)
+
     if sys.platform == 'win32':
         # Unicode arguments are not supported on Windows yet; see #565 and #1074.
         argv_encoding = 'ascii'
hunk ./src/allmydata/util/stringutils.py 139
                                  (output_encoding, repr(s)))
     return out
 
-def quote_output(s, quotemarks=True):
+def quote_output(s, quotemarks=True, encoding=None):
     """
     Encode either a Unicode string or a UTF-8-encoded bytestring for representation
     on stdout or stderr, tolerating errors. If 'quotemarks' is True, the string is
hunk ./src/allmydata/util/stringutils.py 155
             return 'b' + repr(s)
 
     try:
-        out = s.encode(output_encoding)
+        out = s.encode(encoding or output_encoding)
     except (UnicodeEncodeError, UnicodeDecodeError):
         return repr(s)
 
}
[stringutils.py: don't NFC-normalize the output of listdir_unicode.
david-sarah@jacaranda.org**20100617015537
 Ignore-this: 93c9b6f3d7c6812a0afa8d9e1b0b4faa
] {
hunk ./src/allmydata/util/stringutils.py 214
     # On other platforms (ie. Unix systems), the byte-level API is used
 
     if is_unicode_platform:
-        dirlist = os.listdir(path)
+        return os.listdir(path)
     else:
hunk ./src/allmydata/util/stringutils.py 216
-        dirlist = listdir_unicode_fallback(path)
-
-    # Normalize the resulting unicode filenames
-    #
-    # This prevents different OSes from generating non-equal unicode strings for
-    # the same filename representation
-    return [unicodedata.normalize('NFC', fname) for fname in dirlist]
+        return listdir_unicode_fallback(path)
 
 def open_unicode(path, mode):
     """
}
[CLI: allow Unicode patterns in exclude option to 'tahoe backup'.
david-sarah@jacaranda.org**20100617033901
 Ignore-this: 9d971129e1c8bae3c1cc3220993d592e
] {
hunk ./src/allmydata/scripts/cli.py 298
     def opt_exclude(self, pattern):
         """Ignore files matching a glob pattern. You may give multiple
         '--exclude' options."""
-        g = pattern.strip()
+        g = argv_to_unicode(pattern).strip()
         if g:
             exclude = self['exclude']
             exclude.add(g)
hunk ./src/allmydata/scripts/cli.py 305
 
     def opt_exclude_from(self, filepath):
         """Ignore file matching glob patterns listed in file, one per
-        line."""
+        line. The file is assumed to be in the argv encoding."""
         try:
             exclude_file = file(filepath)
         except:
hunk ./src/allmydata/scripts/tahoe_backup.py 176
             children = []
 
         for child in self.options.filter_listdir(children):
+            assert isinstance(child, unicode), child
             childpath = os.path.join(localpath, child)
hunk ./src/allmydata/scripts/tahoe_backup.py 178
-            child = unicode(child)
             # note: symlinks to directories are both islink() and isdir()
             if os.path.isdir(childpath) and not os.path.islink(childpath):
                 metadata = get_local_metadata(childpath)
}
[test_dirnode.py: partial tests for normalization changes.
david-sarah@jacaranda.org**20100617034025
 Ignore-this: 2e3169dd8b120d42dff35bd267dcb417
] {
hunk ./src/allmydata/test/test_dirnode.py 49
 future_write_uri = "x-tahoe-crazy://I_am_from_the_future."
 future_read_uri = "x-tahoe-crazy-readonly://I_am_from_the_future."
 
+# 'o' 'n' 'e-macron'
+one_nfc = u"on\u0113"
+one_nfd = u"one\u0304"
+
 class Dirnode(GridTestMixin, unittest.TestCase,
               testutil.ShouldFailMixin, testutil.StallMixin, ErrorMixin):
     timeout = 240 # It takes longer than 120 seconds on Francois's arm box.
hunk ./src/allmydata/test/test_dirnode.py 80
         c = self.g.clients[0]
         nm = c.nodemaker
 
-        kids = {u"one": (nm.create_from_cap(one_uri), {}),
+        kids = {one_nfd: (nm.create_from_cap(one_uri), {}),
                 u"two": (nm.create_from_cap(setup_py_uri),
                          {"metakey": "metavalue"}),
                 u"mut": (nm.create_from_cap(mut_write_uri, mut_read_uri), {}),
hunk ./src/allmydata/test/test_dirnode.py 105
         
         def _check_kids(children):
             self.failUnlessEqual(set(children.keys()),
-                                 set([u"one", u"two", u"mut", u"fut", u"fro", u"empty_litdir", u"tiny_litdir"]))
-            one_node, one_metadata = children[u"one"]
+                                 set([one_nfc, u"two", u"mut", u"fut", u"fro", u"empty_litdir", u"tiny_litdir"]))
+            one_node, one_metadata = children[one_nfc]
             two_node, two_metadata = children[u"two"]
             mut_node, mut_metadata = children[u"mut"]
             fut_node, fut_metadata = children[u"fut"]
hunk ./src/allmydata/test/test_dirnode.py 161
         d.addCallback(_check_kids)
 
         bad_future_node = UnknownNode(future_write_uri, None)
-        bad_kids1 = {u"one": (bad_future_node, {})}
+        bad_kids1 = {one_nfd: (bad_future_node, {})}
         # This should fail because we don't know how to diminish the future_write_uri
         # cap (given in a write slot and not prefixed with "ro." or "imm.") to a readcap.
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_dirnode.py 169
                                       "cannot attach unknown",
                                       nm.create_new_mutable_directory,
                                       bad_kids1))
-        bad_kids2 = {u"one": (nm.create_from_cap(one_uri), None)}
+        bad_kids2 = {one_nfd: (nm.create_from_cap(one_uri), None)}
         d.addCallback(lambda ign:
                       self.shouldFail(AssertionError, "bad_kids2",
                                       "requires metadata to be a dict",
hunk ./src/allmydata/test/test_dirnode.py 183
         c = self.g.clients[0]
         nm = c.nodemaker
 
-        kids = {u"one": (nm.create_from_cap(one_uri), {}),
+        kids = {one_nfd: (nm.create_from_cap(one_uri), {}),
                 u"two": (nm.create_from_cap(setup_py_uri),
                          {"metakey": "metavalue"}),
                 u"fut": (nm.create_from_cap(None, future_read_uri), {}),
hunk ./src/allmydata/test/test_dirnode.py 209
         
         def _check_kids(children):
             self.failUnlessEqual(set(children.keys()),
-                                 set([u"one", u"two", u"fut", u"empty_litdir", u"tiny_litdir"]))
-            one_node, one_metadata = children[u"one"]
+                                 set([one_nfc, u"two", u"fut", u"empty_litdir", u"tiny_litdir"]))
+            one_node, one_metadata = children[one_nfc]
             two_node, two_metadata = children[u"two"]
             fut_node, fut_metadata = children[u"fut"]
             emptylit_node, emptylit_metadata = children[u"empty_litdir"]
hunk ./src/allmydata/test/test_dirnode.py 254
         d.addCallback(_check_kids)
 
         bad_future_node1 = UnknownNode(future_write_uri, None)
-        bad_kids1 = {u"one": (bad_future_node1, {})}
+        bad_kids1 = {one_nfd: (bad_future_node1, {})}
         d.addCallback(lambda ign:
                       self.shouldFail(MustNotBeUnknownRWError, "bad_kids1",
                                       "cannot attach unknown",
hunk ./src/allmydata/test/test_dirnode.py 261
                                       c.create_immutable_dirnode,
                                       bad_kids1))
         bad_future_node2 = UnknownNode(future_write_uri, future_read_uri)
-        bad_kids2 = {u"one": (bad_future_node2, {})}
+        bad_kids2 = {one_nfd: (bad_future_node2, {})}
         d.addCallback(lambda ign:
                       self.shouldFail(MustBeDeepImmutableError, "bad_kids2",
                                       "is not immutable",
hunk ./src/allmydata/test/test_dirnode.py 267
                                       c.create_immutable_dirnode,
                                       bad_kids2))
-        bad_kids3 = {u"one": (nm.create_from_cap(one_uri), None)}
+        bad_kids3 = {one_nfd: (nm.create_from_cap(one_uri), None)}
         d.addCallback(lambda ign:
                       self.shouldFail(AssertionError, "bad_kids3",
                                       "requires metadata to be a dict",
hunk ./src/allmydata/test/test_dirnode.py 273
                                       c.create_immutable_dirnode,
                                       bad_kids3))
-        bad_kids4 = {u"one": (nm.create_from_cap(mut_write_uri), {})}
+        bad_kids4 = {one_nfd: (nm.create_from_cap(mut_write_uri), {})}
         d.addCallback(lambda ign:
                       self.shouldFail(MustBeDeepImmutableError, "bad_kids4",
                                       "is not immutable",
hunk ./src/allmydata/test/test_dirnode.py 279
                                       c.create_immutable_dirnode,
                                       bad_kids4))
-        bad_kids5 = {u"one": (nm.create_from_cap(mut_read_uri), {})}
+        bad_kids5 = {one_nfd: (nm.create_from_cap(mut_read_uri), {})}
         d.addCallback(lambda ign:
                       self.shouldFail(MustBeDeepImmutableError, "bad_kids5",
                                       "is not immutable",
hunk ./src/allmydata/test/test_dirnode.py 336
             d.addCallback(_check_kids)
             d.addCallback(lambda ign: n.get(u"subdir"))
             d.addCallback(lambda sd: self.failIf(sd.is_mutable()))
-            bad_kids = {u"one": (nm.create_from_cap(mut_write_uri), {})}
+            bad_kids = {one_nfd: (nm.create_from_cap(mut_write_uri), {})}
             d.addCallback(lambda ign:
                           self.shouldFail(MustBeDeepImmutableError, "YZ",
                                           "is not immutable",
}
[test_stringutils.py: take account of the output of listdir_unicode no longer being normalized. Also use Unicode escapes, not UTF-8.
david-sarah@jacaranda.org**20100617034409
 Ignore-this: 47f3f072f0e2efea0abeac130f84c56f
] {
hunk ./src/allmydata/test/test_stringutils.py 1
-# coding=utf-8
+
+lumiere_nfc = u"lumi\u00E8re"
+Artonwall_nfc = u"\u00C4rtonwall.mp3"
+Artonwall_nfd = u"A\u0308rtonwall.mp3"
 
 TEST_FILENAMES = (
hunk ./src/allmydata/test/test_stringutils.py 7
-  u'Ärtonwall.mp3',
+  Artonwall_nfc,
   u'test_file',
   u'Blah blah.txt',
 )
hunk ./src/allmydata/test/test_stringutils.py 22
     import platform
 
     if len(sys.argv) != 2:
-        print "Usage: %s lumière" % sys.argv[0]
+        print "Usage: %s lumi<e-grave>re" % sys.argv[0]
         sys.exit(1)
     
     print
hunk ./src/allmydata/test/test_stringutils.py 62
 from allmydata.util.stringutils import argv_to_unicode, unicode_to_url, \
     unicode_to_output, unicode_platform, listdir_unicode, open_unicode, \
     FilenameEncodingError, get_output_encoding, _reload
+from allmydata.dirnode import normalize
 
 from twisted.python import usage
 
hunk ./src/allmydata/test/test_stringutils.py 96
 
         self.failUnlessRaises(usage.UsageError,
                               argv_to_unicode,
-                              u'lumière'.encode('latin1'))
+                              lumiere_nfc.encode('latin1'))
 
     @patch('sys.stdout')
     def test_unicode_to_output(self, mock):
hunk ./src/allmydata/test/test_stringutils.py 100
-        # Encoding koi8-r cannot represent 'è'
+        # Encoding koi8-r cannot represent e-grave
         mock.encoding = 'koi8-r'
         _reload()
hunk ./src/allmydata/test/test_stringutils.py 103
-        self.failUnlessRaises(UnicodeEncodeError, unicode_to_output, u'lumière')
+        self.failUnlessRaises(UnicodeEncodeError, unicode_to_output, lumiere_nfc)
 
     @patch('os.listdir')
hunk ./src/allmydata/test/test_stringutils.py 106
-    def test_unicode_normalization(self, mock):
-        # Pretend to run on an Unicode platform
+    def test_no_unicode_normalization(self, mock):
+        # Pretend to run on a Unicode platform.
+        # We normalized to NFC in 1.7beta, but we now don't.
         orig_platform = sys.platform
         try:
             sys.platform = 'darwin'
hunk ./src/allmydata/test/test_stringutils.py 112
-            mock.return_value = [u'A\u0308rtonwall.mp3']
+            mock.return_value = [Artonwall_nfd]
             _reload()
hunk ./src/allmydata/test/test_stringutils.py 114
-            self.failUnlessReallyEqual(listdir_unicode(u'/dummy'), [u'\xc4rtonwall.mp3'])
+            self.failUnlessReallyEqual(listdir_unicode(u'/dummy'), [Artonwall_nfd])
         finally:
             sys.platform = orig_platform
 
hunk ./src/allmydata/test/test_stringutils.py 136
         # What happens if latin1-encoded filenames are encountered on an UTF-8
         # filesystem?
         mock_listdir.return_value = [
-            u'lumière'.encode('utf-8'),
-            u'lumière'.encode('latin1')]
+            lumiere_nfc.encode('utf-8'),
+            lumiere_nfc.encode('latin1')]
 
         mock_getfilesystemencoding.return_value = 'utf-8'
         _reload()
hunk ./src/allmydata/test/test_stringutils.py 151
         _reload()
         self.failUnlessRaises(FilenameEncodingError,
                               listdir_unicode,
-                              u'/lumière')
+                              u'/' + lumiere_nfc)
 
     @patch('sys.getfilesystemencoding')
     def test_open_unicode(self, mock):
hunk ./src/allmydata/test/test_stringutils.py 159
         _reload()
         self.failUnlessRaises(FilenameEncodingError,
                               open_unicode,
-                              u'lumière', 'rb')
+                              lumiere_nfc, 'rb')
 
 class StringUtils(ReallyEqualMixin):
     def setUp(self):
hunk ./src/allmydata/test/test_stringutils.py 177
             return
 
         mock.encoding = self.output_encoding
-        argu = u'lumière'
+        argu = lumiere_nfc
         argv = self.argv
         _reload()
         self.failUnlessReallyEqual(argv_to_unicode(argv), argu)
hunk ./src/allmydata/test/test_stringutils.py 183
 
     def test_unicode_to_url(self):
-        self.failUnless(unicode_to_url(u'lumière'), "lumi\xc3\xa8re")
+        self.failUnless(unicode_to_url(lumiere_nfc), "lumi\xc3\xa8re")
 
     @patch('sys.stdout')
     def test_unicode_to_output(self, mock):
hunk ./src/allmydata/test/test_stringutils.py 192
 
         mock.encoding = self.output_encoding
         _reload()
-        self.failUnlessReallyEqual(unicode_to_output(u'lumière'), self.output)
+        self.failUnlessReallyEqual(unicode_to_output(lumiere_nfc), self.output)
 
     def test_unicode_platform(self):
         matrix = {
hunk ./src/allmydata/test/test_stringutils.py 224
         _reload()
         filenames = listdir_unicode(u'/dummy')
 
-        for fname in TEST_FILENAMES:
-            self.failUnless(isinstance(fname, unicode))
-            self.failUnlessIn(fname, filenames)
+        self.failUnlessEqual(set([normalize(fname) for fname in filenames]),
+                             set(TEST_FILENAMES))
 
     @patch('sys.getfilesystemencoding')
     @patch('__builtin__.open')
hunk ./src/allmydata/test/test_stringutils.py 231
     def test_open_unicode(self, mock_open, mock_getfilesystemencoding):
         mock_getfilesystemencoding.return_value = self.filesystem_encoding
-        fn = u'/dummy_directory/lumière.txt'
+        fn = u'/dummy_directory/" + lumiere_nfc + ".txt'
 
         try:
             u"test".encode(self.filesystem_encoding)
}
[stringutils.py: remove unused import.
david-sarah@jacaranda.org**20100617034440
 Ignore-this: 16ec7d737c34665156c2ac486acd545a
] hunk ./src/allmydata/util/stringutils.py 9
 import sys
 import os
 import re
-import unicodedata
 from allmydata.util.assertutil import precondition
 from twisted.python import usage
 import locale
[dirnode.py: comments about normalization changes.
david-sarah@jacaranda.org**20100617041411
 Ignore-this: 9040c4854e73a71dbbb55b50ea3b41b2
] {
hunk ./src/allmydata/dirnode.py 79
 
     return metadata
 
+
+# 'x' at the end of a variable name indicates that it holds a Unicode string that may not
+# be NFC-normalized.
+
 def normalize(namex):
     return unicodedata.normalize('NFC', namex)
 
hunk ./src/allmydata/dirnode.py 332
                 raise ValueError("the rwcapdata field of a dirnode in an immutable directory was not empty")
 
             # A name containing characters that are unassigned in one version of Unicode might
-            # not be normalized wrt a later version. Therefore we normalize names going both in
-            # and out of directories.
+            # not be normalized wrt a later version. See the note in section 'Normalization Stability'
+            # at <http://unicode.org/policies/stability_policy.html>.
+            # Therefore we normalize names going both in and out of directories.
             name = normalize(namex_utf8.decode("utf-8"))
 
             rw_uri = ""
}

Context:

[running.html: fix overeager replacement of 'tahoe' with 'Tahoe-LAFS', and some simplifications.
david-sarah@jacaranda.org**20100617000952
 Ignore-this: 472b4b531c866574ed79f076b58495b5
] 
[Add a specification for servers of happiness.
Kevan Carstensen <kevan@isnotajoke.com>**20100524003508
 Ignore-this: 982e2be8a411be5beaf3582bdfde6151
] 
[Note that servers of happiness only applies to immutable files for the moment
Kevan Carstensen <kevan@isnotajoke.com>**20100524042836
 Ignore-this: cf83cac7a2b3ed347ae278c1a7d9a176
] 
[Add a note about running Tahoe-LAFS on a small grid to running.html
zooko@zooko.com**20100616140227
 Ignore-this: 14dfbff0d47144f7c2375108c6055dc2
 also Change "tahoe" and "Tahoe" to "Tahoe-LAFS" in running.html
 author: Kevan Carstensen
] 
[CLI.txt: introduce 'create-alias' before 'add-alias', document Unicode argument support, and other minor updates.
david-sarah@jacaranda.org**20100610225547
 Ignore-this: de7326e98d79291cdc15aed86ae61fe8
] 
[test_system.py: investigate failure in allmydata.test.test_system.SystemTest.test_upload_and_download_random_key due to bytes_sent not being an int
david-sarah@jacaranda.org**20100616001648
 Ignore-this: 9c78092ab7bfdc909acae3a144ddd1f8
] 
[SFTP: remove a dubious use of 'pragma: no cover'.
david-sarah@jacaranda.org**20100613164356
 Ignore-this: 8f96a81b1196017ed6cfa1d914e56fa5
] 
[SFTP: test that renaming onto a just-opened file fails.
david-sarah@jacaranda.org**20100612033709
 Ignore-this: 9b14147ad78b16a5ab0e0e4813491414
] 
[SFTP: further small improvements to test coverage. Also ensure that after a test failure, later tests don't fail spuriously due to the checks for heisenfile leaks.
david-sarah@jacaranda.org**20100612030737
 Ignore-this: 4ec1dd3d7542be42007987a2f51508e7
] 
[SFTP: further improve test coverage (paths containing '.', bad data for posix-rename extension, and error in test of openShell).
david-sarah@jacaranda.org**20100611213142
 Ignore-this: 956f9df7f9e8a66b506ca58dd9a5dbe7
] 
[SFTP: improve test coverage for no-write on mutable files, and check for heisenfile table leaks in all relevant tests. Delete test_memory_leak since it is now redundant.
david-sarah@jacaranda.org**20100611205752
 Ignore-this: 88be1cf323c10dd534a4b8fdac121e31
] 
[SFTP: add test for extension of file opened with FXF_APPEND.
david-sarah@jacaranda.org**20100610182647
 Ignore-this: c0216d26453ce3cb4b92eef37d218fb4
] 
[NEWS: add UTF-8 coding declaration.
david-sarah@jacaranda.org**20100609234851
 Ignore-this: 3e6ef125b278e0a982c88d23180a78ae
] 
[tests: bump up the timeout on this iputil test from 2s to 4s
zooko@zooko.com**20100609143017
 Ignore-this: 786b7f7bbc85d45cdf727a6293750798
] 
[docs: a few tweaks to NEWS and CREDITS and make quickstart.html point to 1.7.0β!
zooko@zooko.com**20100609142927
 Ignore-this: f8097d3062f41f06c4420a7c84a56481
] 
[docs: Update NEWS file with new features and bugfixes in 1.7.0
francois@ctrlaltdel.ch**20100609091120
 Ignore-this: 8c1014e4469ef530e5ff48d7d6ae71c5
] 
[docs: wording fix, thanks to Jeremy Visser, fix #987
francois@ctrlaltdel.ch**20100609081103
 Ignore-this: 6d2e627e0f1cd58c0e1394e193287a4b
] 
[SFTP: fix most significant memory leak described in #1045 (due to a file being added to all_heisenfiles under more than one direntry when renamed).
david-sarah@jacaranda.org**20100609080003
 Ignore-this: 490b4c14207f6725d0dd32c395fbcefa
] 
[test_stringutils.py: Fix test failure on CentOS builder, possibly Python 2.4.3-related.
david-sarah@jacaranda.org**20100609065056
 Ignore-this: 503b561b213baf1b92ae641f2fdf080a
] 
[docs: update relnote.txt for Tahoe-LAFS v1.7.0β
zooko@zooko.com**20100609054602
 Ignore-this: 52e1bf86a91d45315960fb8806b7a479
] 
[setup: move the mock library from install_requires to tests_require (re: #1016)
zooko@zooko.com**20100609050542
 Ignore-this: c51a4ff3e19ed630755be752d2233db4
] 
[setup: show-tool-versions.py: print out the output from the unix command "locale" and re-arrange encoding data a little bit
zooko@zooko.com**20100609040714
 Ignore-this: 69382719b462d13ff940fcd980776004
] 
[setup: add zope.interface to the packages described by show-tool-versions.py
zooko@zooko.com**20100609034915
 Ignore-this: b5262b2af5c953a5f68a60bd48dcaa75
] 
[Fix for Unicode-related test failures on Zooko's OS X 10.6 machine.
david-sarah@jacaranda.org**20100609055448
 Ignore-this: 395ad16429e56623edfa74457a121190
] 
[stringutils.py, sftpd.py: Portability fixes for Python <= 2.5.
david-sarah@jacaranda.org**20100609013302
 Ignore-this: 9d9ce476ee1b96796e0f48cc5338f852
] 
[_auto_deps.py: allow Python 2.4.3 on Redhat-based distributions.
david-sarah@jacaranda.org**20100609003646
 Ignore-this: ad3cafdff200caf963024873d0ebff3c
] 
[CREDITS: update François's Description
zooko@zooko.com**20100608155513
 Ignore-this: a266b438d25ca2cb28eafff75aa4b2a
] 
[CREDITS: jsgf
zooko@zooko.com**20100608143052
 Ignore-this: 10abe06d40b88e22a9107d30f1b84810
] 
[setup: rename the setuptools_trial .egg that comes bundled in the base dir to not have "-py2.6" in its name, since it works with other versions of python as well
zooko@zooko.com**20100608041607
 Ignore-this: 64fe386d2e5fba0ab441116e74dad5a3
] 
[setup: rename the darcsver .egg that comes bundled in the base dir to not have "-py2.6" in its name, since it works with other versions of python as well
zooko@zooko.com**20100608041534
 Ignore-this: 53f925f160256409cf01b76d2583f83f
] 
[Back out Windows-specific Unicode argument support for v1.7.
david-sarah@jacaranda.org**20100609000803
 Ignore-this: b230ffe6fdaf9a0d85dfe745b37b42fb
] 
[SFTP: suppress NoSuchChildError if heisenfile attributes have been updated in setAttrs, in the case where the parent is available.
david-sarah@jacaranda.org**20100608063753
 Ignore-this: 8c72a5a9c15934f8fe4594ba3ee50ddd
] 
[SFTP: ignore permissions when opening a file (needed for sshfs interoperability).
david-sarah@jacaranda.org**20100608055700
 Ignore-this: f87f6a430f629326a324ddd94426c797
] 
[tests: bump up the timeout on these tests; MM's buildslave is sometimes extremely slow on tests, but it will complete them if given enough time. MM is working on making that buildslave more predictable in how long it takes to run tests.
zooko@zooko.com**20100608033754
 Ignore-this: 98dc27692c5ace1e4b0650b6680629d7
] 
[test_web.py: fix pyflakes warnings introduced by byterange patch.
david-sarah@jacaranda.org**20100608042012
 Ignore-this: a7612724893b51d1154dec4372e0508
] 
[Improve HTTP/1.1 byterange handling
Jeremy Fitzhardinge <jeremy@goop.org>**20100310025913
 Ignore-this: 6d69e694973d618f0dc65983735cd9be
 
 Fix parsing of a Range: header to support:
  - multiple ranges (parsed, but not returned)
  - suffix byte ranges ("-2139")
  - correct handling of incorrectly formatted range headers
    (correct behaviour is to ignore the header and return the full
     file)
  - return appropriate error for ranges outside the file
 
 Multiple ranges are parsed, but only the first range is returned.
 Returning multiple ranges requires using the multipart/byterange
 content type.
 
] 
[test_cli.py: remove invalid 'test_listdir_unicode_bad' test.
david-sarah@jacaranda.org**20100607183730
 Ignore-this: fadfe87980dc1862f349bfcc21b2145f
] 
[check_memory.py: adapt to servers-of-happiness changes.
david-sarah@jacaranda.org**20100608013528
 Ignore-this: c6b28411c543d1aea2f148a955f7998
] 
[show-tool-versions.py: platform.linux_distribution() is not always available
david-sarah@jacaranda.org**20100608004523
 Ignore-this: 793fb4050086723af05d06bed8b1b92a
] 
[show-tool-versions.py: show platform.linux_distribution()
david-sarah@jacaranda.org**20100608003829
 Ignore-this: 81cb5e5fc6324044f0fc6d82903c8223
] 
[Remove the 'tahoe debug consolidate' subcommand.
david-sarah@jacaranda.org**20100607183757
 Ignore-this: 4b14daa3ae557cea07d6e119d25dafe9
] 
[tests: drastically increase timeout of this very time-consuming test in honor of François's ARM box
zooko@zooko.com**20100607115929
 Ignore-this: bf1bb52ffb6b5ccae71d4dde14621bc8
] 
[setup: update authorship, datestamp, licensing, and add special exceptions to allow combination with Eclipse- and QPL- licensed code
zooko@zooko.com**20100607062329
 Ignore-this: 5a1d7b12dfafd61283ea65a245416381
] 
[common_http.py, tahoe_cp.py: Fix an error in calling the superclass constructor in HTTPError and MissingSourceError (introduced by the Unicode fixes).
david-sarah@jacaranda.org**20100607174714
 Ignore-this: 1a118d593d81c918a4717c887f033aec
] 
[FTP-and-SFTP.txt: minor technical correction to doc for 'no-write' flag.
david-sarah@jacaranda.org**20100607061600
 Ignore-this: 66aee0c1b6c00538602d08631225e114
] 
[test_stringutils.py: trivial error in exception message for skipped test.
david-sarah@jacaranda.org**20100607061455
 Ignore-this: f261a5d4e2b8fe3bcc37e02539ba1ae2
] 
[setup: organize misc/ scripts and tools and remove obsolete ones
zooko@zooko.com**20100607051618
 Ignore-this: 161db1158c6b7be8365b0b3dee2e0b28
 This is for ticket #1068.
] 
[More Unicode test fixes.
david-sarah@jacaranda.org**20100607053358
 Ignore-this: 6a271fb77c31f28cb7bdba63b26a2dd2
] 
[Unicode fixes for platforms with non-native-Unicode filesystems.
david-sarah@jacaranda.org**20100607043238
 Ignore-this: 2134dc1793c4f8e50350bd749c4c98c2
] 
[Unicode fixes.
david-sarah@jacaranda.org**20100607010215
 Ignore-this: d58727b5cd2ce00e6b6dae3166030138
] 
[quickstart.html: link to snapshots page, sorted with most recent first.
david-sarah@jacaranda.org**20100606221127
 Ignore-this: 93ea7e6ee47acc66f6daac9cabffed2d
] 
[setup: loosen the Desert Island test to allow it to check the network for new packages as long as it doesn't actually download any
zooko@zooko.com**20100606175717
 Ignore-this: e438a8eb3c1b0e68080711ec6ff93ffa
 (You can look but don't touch.)
] 
[setup: have the buildbots print out locale.getpreferredencoding(), locale.getdefaultlocale(), locale.getlocale(), and os.path.supports_unicode_filenames
zooko@zooko.com**20100605162932
 Ignore-this: 85e31e0e0e1364e9215420e272d58116
 Even though that latter one is completely useless, I'm curious.
] 
[quickstart.html: We haven't released 1.7beta yet.
david-sarah@jacaranda.org**20100606220301
 Ignore-this: 4e18898cfdb08cc3ddd1ff94d43fdda7
] 
[Raise Python version requirement to 2.4.4 for non-UCS-2 builds, to avoid a critical Python security bug.
david-sarah@jacaranda.org**20100605031713
 Ignore-this: 2df2b6d620c5d8191c79eefe655059e2
] 
[unicode tests: fix missing import
zooko@zooko.com**20100604142630
 Ignore-this: db437fe8009971882aaea9de05e2bc3
] 
[unicode: make test_cli test a non-ascii argument, and make the fallback term encoding be locale.getpreferredencoding()
zooko@zooko.com**20100604141251
 Ignore-this: b2bfc07942f69141811e59891842bd8c
] 
[unicode: always decode json manifest as utf-8 then encode for stdout
zooko@zooko.com**20100604084840
 Ignore-this: ac481692315fae870a0f3562bd7db48e
 pyflakes pointed out that the exception handler fallback called an un-imported function, showing that the fallback wasn't being exercised.
 I'm not 100% sure that this patch is right and would appreciate François or someone reviewing it.
] 
[fix flakes
zooko@zooko.com**20100604075845
 Ignore-this: 3e6a84b78771b0ad519e771a13605f0
] 
[fix syntax of assertion handling that isn't portable to older versions of Python
zooko@zooko.com**20100604075805
 Ignore-this: 3a12b293aad25883fb17230266eb04ec
] 
[test_stringutils.py: Skip test test_listdir_unicode_good if filesystem supports only ASCII filenames
Francois Deppierraz <francois@ctrlaltdel.ch>**20100521160839
 Ignore-this: f2ccdbd04c8d9f42f1efb0eb80018257
] 
[test_stringutils.py: Skip test_listdir_unicode on mocked platform which cannot store non-ASCII filenames
Francois Deppierraz <francois@ctrlaltdel.ch>**20100521160559
 Ignore-this: b93fde736a8904712b506e799250a600
] 
[test_stringutils.py: Add a test class for OpenBSD 4.1 with LANG=C
Francois Deppierraz <francois@ctrlaltdel.ch>**20100521140053
 Ignore-this: 63f568aec259cef0e807752fc8150b73
] 
[test_stringutils.py: Mock the open() call in test_open_unicode
Francois Deppierraz <francois@ctrlaltdel.ch>**20100521135817
 Ignore-this: d8be4e56a6eefe7d60f97f01ea20ac67
 
 This test ensure that open(a_unicode_string) is used on Unicode platforms
 (Windows or MacOS X) and that open(a_correctly_encoded_bytestring) on other
 platforms such as Unix.
 
] 
[test_stringutils.py: Fix a trivial Python 2.4 syntax incompatibility
Francois Deppierraz <francois@ctrlaltdel.ch>**20100521093345
 Ignore-this: 9297e3d14a0dd37d0c1a4c6954fd59d3
] 
[test_cli.py: Fix tests when sys.stdout.encoding=None and refactor this code into functions
Francois Deppierraz <francois@ctrlaltdel.ch>**20100520084447
 Ignore-this: cf2286e225aaa4d7b1927c78c901477f
] 
[Fix handling of correctly encoded unicode filenames (#534)
Francois Deppierraz <francois@ctrlaltdel.ch>**20100520004356
 Ignore-this: 8a3a7df214a855f5a12dc0eeab6f2e39
 
 Tahoe CLI commands working on local files, for instance 'tahoe cp' or 'tahoe
 backup', have been improved to correctly handle filenames containing non-ASCII
 characters.
   
 In the case where Tahoe encounters a filename which cannot be decoded using the
 system encoding, an error will be returned and the operation will fail.  Under
 Linux, this typically happens when the filesystem contains filenames encoded
 with another encoding, for instance latin1, than the system locale, for
 instance UTF-8.  In such case, you'll need to fix your system with tools such
 as 'convmv' before using Tahoe CLI.
   
 All CLI commands have been improved to support non-ASCII parameters such as
 filenames and aliases on all supported Operating Systems except Windows as of
 now.
] 
[stringutils.py: Unicode helper functions + associated tests
Francois Deppierraz <francois@ctrlaltdel.ch>**20100520004105
 Ignore-this: 7a73fc31de2fd39d437d6abd278bfa9a
 
 This file contains a bunch of helper functions which converts
 unicode string from and to argv, filenames and stdout.
] 
[Add dependency on Michael Foord's mock library
Francois Deppierraz <francois@ctrlaltdel.ch>**20100519233325
 Ignore-this: 9bb01bf1e4780f6b98ed394c3b772a80
] 
[setup: adjust make clean target to ignore our bundled build tools
zooko@zooko.com**20100604051250
 Ignore-this: d24d2a3b849000790cfbfab69237454e
] 
[setup: bundle a copy of setuptools_trial as an unzipped egg in the base dir of the Tahoe-LAFS source tree
zooko@zooko.com**20100604044648
 Ignore-this: a4736e9812b4dab2d5a2bc4bfc5c3b28
 This is to work-around this Distribute issue:
 http://bitbucket.org/tarek/distribute/issue/55/revision-control-plugin-automatically-installed-as-a-build-dependency-is-not-present-when-another-build-dependency-is-being
] 
[setup: bundle a copy of darcsver in unzipped egg form in the root of the Tahoe-LAFS source tree
zooko@zooko.com**20100604044146
 Ignore-this: a51a52e82dd3a39225657ffa27decae2
 This is to work-around this Distribute issue:
 http://bitbucket.org/tarek/distribute/issue/55/revision-control-plugin-automatically-installed-as-a-build-dependency-is-not-present-when-another-build-dependency-is-being
] 
[setup: undo the previous patch to quote the executable in scripts
zooko@zooko.com**20100604025204
 Ignore-this: beda3b951c49d1111478618b8cabe005
 The problem isn't in the script, it is in the cli.exe script that is built by setuptools. This might be related to
 http://bugs.python.org/issue6792
 and
 http://bugs.python.org/setuptools/issue2
 Or it might be a separate issue involving the launcher.c code e.g. http://tahoe-lafs.org/trac/zetuptoolz/browser/launcher.c?rev=576#L210 and its handling of the interpreter name.
] 
[quickstart.html: warn against installing Python at a path containing spaces.
david-sarah@jacaranda.org**20100604032413
 Ignore-this: c7118332573abd7762d9a897e650bc6a
] 
[setup: put quotes around the path to executable in case it has spaces in it, when building a tahoe.exe for win32
zooko@zooko.com**20100604020836
 Ignore-this: 478684843169c94a9c14726fedeeed7d
] 
[misc/show-tool-versions.py: Display additional Python interpreter encoding informations (stdout, stdin and filesystem)
Francois Deppierraz <francois@ctrlaltdel.ch>**20100521094313
 Ignore-this: 3ae9b0b07fd1d53fb632ef169f7c5d26
] 
[Fix test failures in test_web caused by changes to web page titles in #1062. Also, change a 'target' field to '_blank' instead of 'blank' in welcome.xhtml.
david-sarah@jacaranda.org**20100603232105
 Ignore-this: 6e2cc63f42b07e2a3b2d1a857abc50a6
] 
[Resolve merge conflict for sftpd.py
david-sarah@jacaranda.org**20100603182537
 Ignore-this: ba8b543e51312ac949798eb8f5bd9d9c
] 
[SFTP: possible fix for metadata times being shown as the epoch.
david-sarah@jacaranda.org**20100602234514
 Ignore-this: bdd7dfccf34eff818ff88aa4f3d28790
] 
[SFTP: further improvements to test coverage.
david-sarah@jacaranda.org**20100602234422
 Ignore-this: 87eeee567e8d7562659442ea491e187c
] 
[SFTP: improve test coverage. Also make creating a directory fail when permissions are read-only (rather than ignoring the permissions).
david-sarah@jacaranda.org**20100602041934
 Ignore-this: a5e9d9081677bc7f3ddb18ca7a1f531f
] 
[dirnode.py: fix a bug in the no-write change for Adder, and improve test coverage. Add a 'metadata' argument to create_subdirectory, with documentation. Also update some comments in test_dirnode.py made stale by the ctime/mtime change.
david-sarah@jacaranda.org**20100602032641
 Ignore-this: 48817b54cd63f5422cb88214c053b03b
] 
[dirnode.py: Fix bug that caused 'tahoe' fields, 'ctime' and 'mtime' not to be updated when new metadata is present.
david-sarah@jacaranda.org**20100602014644
 Ignore-this: 5bac95aa897b68f2785d481e49b6a66
] 
[SFTP: fix a bug that caused the temporary files underlying EncryptedTemporaryFiles not to be closed.
david-sarah@jacaranda.org**20100601055310
 Ignore-this: 44fee4cfe222b2b1690f4c5e75083a52
] 
[SFTP: changes for #1063 ('no-write' field) including comment:1 (clearing owner write permission diminishes to a read cap). Includes documentation changes, but not tests for the new behaviour.
david-sarah@jacaranda.org**20100601051139
 Ignore-this: eff7c08bd47fd52bfe2b844dabf02558
] 
[dirnode.py: Fix #1034 (MetadataSetter does not enforce restriction on setting 'tahoe' subkeys), and expose the metadata updater for use by SFTP. Also, support diminishing a child cap to read-only if 'no-write' is set in the metadata.
david-sarah@jacaranda.org**20100601045428
 Ignore-this: 14f26e17e58db97fad0dcfd350b38e95
] 
[SFTP: the same bug as in _sync_heisenfiles also occurred in two other places.
david-sarah@jacaranda.org**20100530060127
 Ignore-this: 8d137658fc6e4596fa42697476c39aa3
] 
[SFTP: another try at fixing the _sync_heisenfiles bug.
david-sarah@jacaranda.org**20100530055254
 Ignore-this: c15f76f32a60083a6b7de6ca0e917934
] 
[SFTP: fix silly bug in _sync_heisenfiles ('f is not ignore' vs 'not (f is ignore)').
david-sarah@jacaranda.org**20100530053807
 Ignore-this: 71c4bc62613bf8fef835886d8eb61c27
] 
[SFTP: log when a sync completes.
david-sarah@jacaranda.org**20100530051840
 Ignore-this: d99765663ceb673c8a693dfcf88c25ea
] 
[SFTP: fix bug in previous logging patch.
david-sarah@jacaranda.org**20100530050000
 Ignore-this: 613e4c115f03fe2d04c621b510340817
] 
[SFTP: more logging to track down OpenOffice hang.
david-sarah@jacaranda.org**20100530040809
 Ignore-this: 6c11f2d1eac9f62e2d0f04f006476a03
] 
[SFTP: avoid blocking close on a heisenfile that has been abandoned or never changed. Also, improve the logging to help track down a case where OpenOffice hangs on opening a file with FXF_READ|FXF_WRITE.
david-sarah@jacaranda.org**20100530025544
 Ignore-this: 9919dddd446fff64de4031ad51490d1c
] 
[Move suppression of DeprecationWarning about BaseException.message from sftpd.py to main __init__.py. Also, remove the global suppression of the 'integer argument expected, got float' warning, which turned out to be a bug.
david-sarah@jacaranda.org**20100529050537
 Ignore-this: 87648afa0dec0d2e73614007de102a16
] 
[SFTP: cater to clients that assume a file is created as soon as they have made an open request; also, fix some race conditions associated with closing a file at about the same time as renaming or removing it.
david-sarah@jacaranda.org**20100529045253
 Ignore-this: 2404076b2154ff2659e2b10e0b9e813c
] 
[Change doc comments in interfaces.py to take into account unknown nodes.
david-sarah@jacaranda.org**20100528171922
 Ignore-this: d2fde6890b3bca9c7275775f64fbff56
] 
[Add must_exist, must_be_directory, and must_be_file arguments to DirectoryNode.delete. This will be used to fixes a minor condition in the SFTP frontend.
david-sarah@jacaranda.org**20100527194529
 Ignore-this: 6d8114cef4450c52c57639f82852716f
] 
[Trivial whitespace changes.
david-sarah@jacaranda.org**20100527194114
 Ignore-this: 98d611bc54ee20b01a5f6b334ff61b2d
] 
[SFTP: 'sync' any open files at a direntry before opening any new file at that direntry. This works around the sshfs misbehaviour of returning success to clients immediately on close.
david-sarah@jacaranda.org**20100525230257
 Ignore-this: 63245d6d864f8f591c86170864d7c57f
] 
[SFTP: handle removing a file while it is open. Also some simplifications of the logout handling.
david-sarah@jacaranda.org**20100525184210
 Ignore-this: 660ee80be6ecab783c60452a9da896de
] 
[SFTP: a posix-rename response should actually return an FXP_STATUS reply, not an FXP_EXTENDED_REPLY as Twisted Conch assumes. Work around this by raising an SFTPError with code FX_OK.
david-sarah@jacaranda.org**20100525033323
 Ignore-this: fe2914d3ef7f5194bbeaf3f2dda2ad7d
] 
[SFTP: fix problem with posix-rename code returning a Deferred for the renamed filenode, not for the result of the request (an empty string).
david-sarah@jacaranda.org**20100525020209
 Ignore-this: 69f7491df2a8f7ea92d999a6d9f0581d
] 
[SFTP: fix time handling to make sure floats are not passed into twisted.conch, and to print times in the future less ambiguously in directory listings.
david-sarah@jacaranda.org**20100524230412
 Ignore-this: eb1a3fb72492fa2fb19667b6e4300440
] 
[SFTP: name of the POSIX rename extension should be 'posix-rename@openssh.com', not 'extposix-rename@openssh.com'.
david-sarah@jacaranda.org**20100524021156
 Ignore-this: f90eb1ff9560176635386ee797a3fdc7
] 
[SFTP: avoid race condition where .write could be called on an OverwriteableFileConsumer after it had been closed.
david-sarah@jacaranda.org**20100523233830
 Ignore-this: 55d381064a15bd64381163341df4d09f
] 
[SFTP: log tracebacks for RAISEd exceptions.
david-sarah@jacaranda.org**20100523221535
 Ignore-this: c76a7852df099b358642f0631237cc89
] 
[Suppress 'integer argument expected, got float' DeprecationWarning everywhere
david-sarah@jacaranda.org**20100523221157
 Ignore-this: 80efd7e27798f5d2ad66c7a53e7048e5
] 
[SFTP: more logging to investigate behaviour of getAttrs(path).
david-sarah@jacaranda.org**20100523204236
 Ignore-this: e58fd35dc9015316e16a9f49f19bb469
] 
[SFTP: fix pyflakes warnings; drop 'noisy' versions of eventually_callback and eventually_errback; robustify conversion of exception messages to UTF-8.
david-sarah@jacaranda.org**20100523140905
 Ignore-this: 420196fc58646b05bbc9c3732b6eb314
] 
[SFTP: fixes and test cases for renaming of open files.
david-sarah@jacaranda.org**20100523032549
 Ignore-this: 32e0726be0fc89335f3035157e202c68
] 
[SFTP: Increase test_sftp timeout to cater for francois' ARM buildslave.
david-sarah@jacaranda.org**20100522191639
 Ignore-this: a5acf9660d304677048ab4dd72908ad8
] 
[SFTP: Fix error in support for getAttrs on an open file, to index open files by directory entry rather than path. Extend that support to renaming open files. Also, implement the extposix-rename@openssh.org extension, and some other minor refactoring.
david-sarah@jacaranda.org**20100522035836
 Ignore-this: 8ef93a828e927cce2c23b805250b81a4
] 
[SFTP: relax pyasn1 version dependency to >= 0.0.8a.
david-sarah@jacaranda.org**20100520181437
 Ignore-this: 2c7b3dee7b7e14ba121d3118193a386a
] 
[SFTP tests: fix test_openDirectory_and_attrs that was failing in timezones west of UTC.
david-sarah@jacaranda.org**20100520181027
 Ignore-this: 9beaf602beef437c11c7e97f54ce2599
] 
[SFTP: allow getAttrs to succeed on a file that has been opened for creation but not yet uploaded or linked (part of #1050).
david-sarah@jacaranda.org**20100520035613
 Ignore-this: 2f59107d60d5476edac19361ccf6cf94
] 
[SFTP: improve logging so that results of requests are (usually) logged.
david-sarah@jacaranda.org**20100520003652
 Ignore-this: 3f59eeee374a3eba71db9be31d5a95
] 
[SFTP: add tests for more combinations of open flags.
david-sarah@jacaranda.org**20100519053933
 Ignore-this: b97ee351b1e8ecfecabac70698060665
] 
[SFTP: allow FXF_WRITE | FXF_TRUNC (#1050).
david-sarah@jacaranda.org**20100519043240
 Ignore-this: bd70009f11d07ac6e9fd0d1e3fa87a9b
] 
[SFTP: remove another case where we were logging data.
david-sarah@jacaranda.org**20100519012713
 Ignore-this: 83115daf3a90278fed0e3fc267607584
] 
[SFTP: avoid logging all data passed to callbacks.
david-sarah@jacaranda.org**20100519000651
 Ignore-this: ade6d69a473ada50acef6389fc7fdf69
] 
[SFTP: fixes related to reporting of permissions (needed for sshfs).
david-sarah@jacaranda.org**20100518054521
 Ignore-this: c51f8a5d0dc76b80d33ffef9b0541325
] 
[SFTP: change error code returned for ExistingChildError to FX_FAILURE (fixes gvfs with some picky programs such as gedit).
david-sarah@jacaranda.org**20100518004205
 Ignore-this: c194c2c9aaf3edba7af84b7413cec375
] 
[SFTP: fixed bugs that caused hangs during write (#1037).
david-sarah@jacaranda.org**20100517044228
 Ignore-this: b8b95e82c4057367388a1e6baada993b
] 
[SFTP: work around a probable bug in twisted.conch.ssh.session:loseConnection(). Also some minor error handling cleanups.
david-sarah@jacaranda.org**20100517012606
 Ignore-this: 5d3da7c4219cb0c14547e7fd70c74204
] 
[SFTP: add pyasn1 as dependency, needed if we are using Twisted >= 9.0.0.
david-sarah@jacaranda.org**20100516193710
 Ignore-this: 76fd92e8a950bb1983a90a09e89c54d3
] 
[SFTP: Support statvfs extensions, avoid logging actual data, and decline shell sessions politely.
david-sarah@jacaranda.org**20100516154347
 Ignore-this: 9d05d23ba77693c03a61accd348ccbe5
] 
[SFTP: fix error in SFTPUserHandler arguments introduced by execCommand patch.
david-sarah@jacaranda.org**20100516014045
 Ignore-this: f5ee494dc6ad6aa536cc8144bd2e3d19
] 
[SFTP: implement execCommand to interoperate with clients that issue a 'df -P -k /' command. Also eliminate use of Zope adaptation.
david-sarah@jacaranda.org**20100516012754
 Ignore-this: 2d0ed28b759f67f83875b1eaf5778992
] 
[sftpd.py: 'log.OPERATIONAL' should be just 'OPERATIONAL'.
david-sarah@jacaranda.org**20100515155533
 Ignore-this: f2347cb3301bbccc086356f6edc685
] 
[Attempt to fix #1040 by making SFTPUser implement ISession.
david-sarah@jacaranda.org**20100515005719
 Ignore-this: b3baaf088ba567e861e61e347195dfc4
] 
[Eliminate Windows newlines from sftpd.py.
david-sarah@jacaranda.org**20100515005656
 Ignore-this: cd54fd25beb957887514ae76e08c277
] 
[Update SFTP implementation and tests: fix #1038 and switch to foolscap logging; also some code reorganization.
david-sarah@jacaranda.org**20100514043113
 Ignore-this: 262f76d953dcd4317210789f2b2bf5da
] 
[Change shouldFail to avoid Unicode errors when converting Failure to str
david-sarah@jacaranda.org**20100512060754
 Ignore-this: 86ed419d332d9c33090aae2cde1dc5df
] 
[Tests for new SFTP implementation
david-sarah@jacaranda.org**20100512060552
 Ignore-this: 20308d4a59b3ebc868aad55ae0a7a981
] 
[New SFTP implementation: mutable files, read/write support, streaming download, Unicode filenames, and more
david-sarah@jacaranda.org**20100512055407
 Ignore-this: 906f51c48d974ba9cf360c27845c55eb
] 
[allmydata.org -> tahoe-lafs.org in __init__.py
david-sarah@jacaranda.org**20100603063530
 Ignore-this: f7d82331d5b4a3c4c0938023409335af
] 
[small change to CREDITS
david-sarah@jacaranda.org**20100603062421
 Ignore-this: 2909cdbedc19da5573dec810fc23243
] 
[Resolve conflict in patch to change imports to absolute.
david-sarah@jacaranda.org**20100603054608
 Ignore-this: 15aa1caa88e688ffa6dc53bed7dcca7d
] 
[Minor documentation tweaks.
david-sarah@jacaranda.org**20100603054458
 Ignore-this: e30ae407b0039dfa5b341d8f88e7f959
] 
[title_rename_xhtml.dpatch.txt
freestorm77@gmail.com**20100529172542
 Ignore-this: d2846afcc9ea72ac443a62ecc23d121b
 
 - Renamed xhtml Title from "Allmydata - Tahoe" to "Tahoe-LAFS"
 - Renamed Tahoe to Tahoe-LAFS in page content
 - Changed Tahoe-LAFS home page link to http://tahoe-lafs.org (added target="blank")
 - Deleted commented css script in info.xhtml
 
 
] 
[tests: refactor test_web.py to have less duplication of literal caps-from-the-future
zooko@zooko.com**20100519055146
 Ignore-this: 49e5412e6cc4566ca67f069ffd850af6
 This is a prelude to a patch which will add tests of caps from the future which have non-ascii chars in them.
] 
[doc_reformat_stats.txt
freestorm77@gmail.com**20100424114615
 Ignore-this: af315db5f7e3a17219ff8fb39bcfcd60
 
 
    - Added heading format begining and ending by "=="
    - Added Index
    - Added Title
           
    Note: No change are made in paragraphs content
 
 
 **END OF DESCRIPTION***
 
 Place the long patch description above the ***END OF DESCRIPTION*** marker.
 The first line of this file will be the patch name.
 
 
 This patch contains the following changes:
 
 M ./docs/stats.txt -2 +2
] 
[doc_reformat_performance.txt
freestorm77@gmail.com**20100424114444
 Ignore-this: 55295ff5cd8a5b67034eb661a5b0699d
 
    - Added heading format begining and ending by "=="
    - Added Index
    - Added Title
         
    Note: No change are made in paragraphs content
 
 
] 
[doc_refomat_logging.txt
freestorm77@gmail.com**20100424114316
 Ignore-this: 593f0f9914516bf1924dfa6eee74e35f
 
    - Added heading format begining and ending by "=="
    - Added Index
    - Added Title
         
    Note: No change are made in paragraphs content
 
] 
[doc_reformat_known_issues.txt
freestorm77@gmail.com**20100424114118
 Ignore-this: 9577c3965d77b7ac18698988cfa06049
 
     - Added heading format begining and ending by "=="
     - Added Index
     - Added Title
           
     Note: No change are made in paragraphs content
   
 
] 
[doc_reformat_helper.txt
freestorm77@gmail.com**20100424120649
 Ignore-this: de2080d6152ae813b20514b9908e37fb
 
 
    - Added heading format begining and ending by "=="
    - Added Index
    - Added Title
             
    Note: No change are made in paragraphs content
 
] 
[doc_reformat_garbage-collection.txt
freestorm77@gmail.com**20100424120830
 Ignore-this: aad3e4c99670871b66467062483c977d
 
 
    - Added heading format begining and ending by "=="
    - Added Index
    - Added Title
             
    Note: No change are made in paragraphs content
 
] 
[doc_reformat_FTP-and-SFTP.txt
freestorm77@gmail.com**20100424121334
 Ignore-this: 3736b3d8f9a542a3521fbb566d44c7cf
 
 
    - Added heading format begining and ending by "=="
    - Added Index
    - Added Title
           
    Note: No change are made in paragraphs content
 
] 
[doc_reformat_debian.txt
freestorm77@gmail.com**20100424120537
 Ignore-this: 45fe4355bb869e55e683405070f47eff
 
 
    - Added heading format begining and ending by "=="
    - Added Index
    - Added Title
             
    Note: No change are made in paragraphs content
 
] 
[doc_reformat_configuration.txt
freestorm77@gmail.com**20100424104903
 Ignore-this: 4fbabc51b8122fec69ce5ad1672e79f2
 
 
 - Added heading format begining and ending by "=="
 - Added Index
 - Added Title
 
 Note: No change are made in paragraphs content
 
] 
[doc_reformat_CLI.txt
freestorm77@gmail.com**20100424121512
 Ignore-this: 2d3a59326810adcb20ea232cea405645
 
      - Added heading format begining and ending by "=="
      - Added Index
      - Added Title
           
      Note: No change are made in paragraphs content
 
] 
[doc_reformat_backupdb.txt
freestorm77@gmail.com**20100424120416
 Ignore-this: fed696530e9d2215b6f5058acbedc3ab
 
 
    - Added heading format begining and ending by "=="
    - Added Index
    - Added Title
             
    Note: No change are made in paragraphs content
 
] 
[doc_reformat_architecture.txt
freestorm77@gmail.com**20100424120133
 Ignore-this: 6e2cab4635080369f2b8cadf7b2f58e
 
 
     - Added heading format begining and ending by "=="
     - Added Index
     - Added Title
             
     Note: No change are made in paragraphs content
 
 
] 
[Correct harmless indentation errors found by pylint
david-sarah@jacaranda.org**20100226052151
 Ignore-this: 41335bce830700b18b80b6e00b45aef5
] 
[Change relative imports to absolute
david-sarah@jacaranda.org**20100226071433
 Ignore-this: 32e6ce1a86e2ffaaba1a37d9a1a5de0e
] 
[Document reason for the trialcoverage version requirement being 0.3.3.
david-sarah@jacaranda.org**20100525004444
 Ignore-this: 2f9f1df6882838b000c063068f258aec
] 
[Downgrade version requirement for trialcoverage to 0.3.3 (from 0.3.10), to avoid needing to compile coveragepy on Windows.
david-sarah@jacaranda.org**20100524233707
 Ignore-this: 9c397a374c8b8017e2244b8a686432a8
] 
[Suppress deprecation warning for twisted.web.error.NoResource when using Twisted >= 9.0.0.
david-sarah@jacaranda.org**20100516205625
 Ignore-this: 2361a3023cd3db86bde5e1af759ed01
] 
[docs: CREDITS for Jeremy Visser
zooko@zooko.com**20100524081829
 Ignore-this: d7c1465fd8d4e25b8d46d38a1793465b
] 
[test: show stdout and stderr in case of non-zero exit code from "tahoe" command
zooko@zooko.com**20100524073348
 Ignore-this: 695e81cd6683f4520229d108846cd551
] 
[setup: upgrade bundled zetuptoolz to zetuptoolz-0.6c15dev and make it unpacked and directly loaded by setup.py
zooko@zooko.com**20100523205228
 Ignore-this: 24fb32aaee3904115a93d1762f132c7
 Also fix the relevant "make clean" target behavior.
] 
[setup: remove bundled zipfile egg of setuptools
zooko@zooko.com**20100523205120
 Ignore-this: c68b5f2635bb93d1c1fa7b613a026f9e
 We're about to replace it with bundled unpacked source code of setuptools, which is much nicer for debugging and evolving under revision control.
] 
[setup: remove bundled copy of setuptools_trial-0.5.2.tar
zooko@zooko.com**20100522221539
 Ignore-this: 140f90eb8fb751a509029c4b24afe647
 Hopefully it will get installed automatically as needed and we won't bundle it anymore.
] 
[setup: remove bundled setuptools_darcs-1.2.8.tar
zooko@zooko.com**20100522015333
 Ignore-this: 378b1964b513ae7fe22bae2d3478285d
 This version of setuptools_darcs had a bug when used on Windows which has been fixed in setuptools_darcs-1.2.9. Hopefully we will not need to bundle a copy of setuptools_darcs-1.2.9 in with Tahoe-LAFS and can instead rely on it to be downloaded from PyPI or bundled in the "tahoe deps" separate tarball.
] 
[tests: fix pyflakes warnings in bench_dirnode.py
zooko@zooko.com**20100521202511
 Ignore-this: f23d55b4ed05e52865032c65a15753c4
] 
[setup: if the string '--reporter=bwverbose-coverage' appears on sys.argv then you need trialcoverage
zooko@zooko.com**20100521122226
 Ignore-this: e760c45dcfb5a43c1dc1e8a27346bdc2
] 
[tests: don't let bench_dirnode.py do stuff and have side-effects at import time (unless __name__ == '__main__')
zooko@zooko.com**20100521122052
 Ignore-this: 96144a412250d9bbb5fccbf83b8753b8
] 
[tests: increase timeout to give François's ARM buildslave a chance to complete the tests
zooko@zooko.com**20100520134526
 Ignore-this: 3dd399fdc8b91149c82b52f955b50833
] 
[run_trial.darcspath
freestorm77@gmail.com**20100510232829
 Ignore-this: 5ebb4df74e9ea8a4bdb22b65373d1ff2
] 
[docs: line-wrap README.txt
zooko@zooko.com**20100518174240
 Ignore-this: 670a02d360df7de51ebdcf4fae752577
] 
[Hush pyflakes warnings
Kevan Carstensen <kevan@isnotajoke.com>**20100515184344
 Ignore-this: fd602c3bba115057770715c36a87b400
] 
[setup: new improved misc/show-tool-versions.py
zooko@zooko.com**20100516050122
 Ignore-this: ce9b1de1b35b07d733e6cf823b66335a
] 
[Improve code coverage of the Tahoe2PeerSelector tests.
Kevan Carstensen <kevan@isnotajoke.com>**20100515032913
 Ignore-this: 793151b63ffa65fdae6915db22d9924a
] 
[Remove a comment that no longer makes sense.
Kevan Carstensen <kevan@isnotajoke.com>**20100514203516
 Ignore-this: 956983c7e7c7e4477215494dfce8f058
] 
[docs: update docs/architecture.txt to more fully and correctly explain the upload procedure
zooko@zooko.com**20100514043458
 Ignore-this: 538b6ea256a49fed837500342092efa3
] 
[Fix up the behavior of #778, per reviewers' comments
Kevan Carstensen <kevan@isnotajoke.com>**20100514004917
 Ignore-this: 9c20b60716125278b5456e8feb396bff
 
   - Make some important utility functions clearer and more thoroughly 
     documented.
   - Assert in upload.servers_of_happiness that the buckets attributes
     of PeerTrackers passed to it are mutually disjoint.
   - Get rid of some silly non-Pythonisms that I didn't see when I first
     wrote these patches.
   - Make sure that should_add_server returns true when queried about a 
     shnum that it doesn't know about yet.
   - Change Tahoe2PeerSelector.preexisting_shares to map a shareid to a set
     of peerids, alter dependencies to deal with that.
   - Remove upload.should_add_servers, because it is no longer necessary
   - Move upload.shares_of_happiness and upload.shares_by_server to a utility
     file.
   - Change some points in Tahoe2PeerSelector.
   - Compute servers_of_happiness using a bipartite matching algorithm that 
     we know is optimal instead of an ad-hoc greedy algorithm that isn't.
   - Change servers_of_happiness to just take a sharemap as an argument,
     change its callers to merge existing_shares and used_peers before 
     calling it.
   - Change an error message in the encoder to be more appropriate for 
     servers of happiness.
   - Clarify the wording of an error message in immutable/upload.py
   - Refactor a happiness failure message to happinessutil.py, and make
     immutable/upload.py and immutable/encode.py use it.
   - Move the word "only" as far to the right as possible in failure 
     messages.
   - Use a better definition of progress during peer selection.
   - Do read-only peer share detection queries in parallel, not sequentially.
   - Clean up logging semantics; print the query statistics whenever an
     upload is unsuccessful, not just in one case.
 
] 
[Alter the error message when an upload fails, per some comments in #778.
Kevan Carstensen <kevan@isnotajoke.com>**20091230210344
 Ignore-this: ba97422b2f9737c46abeb828727beb1
 
 When I first implemented #778, I just altered the error messages to refer to
 servers where they referred to shares. The resulting error messages weren't
 very good. These are a bit better.
] 
[Change "UploadHappinessError" to "UploadUnhappinessError"
Kevan Carstensen <kevan@isnotajoke.com>**20091205043037
 Ignore-this: 236b64ab19836854af4993bb5c1b221a
] 
[Alter the error message returned when peer selection fails
Kevan Carstensen <kevan@isnotajoke.com>**20091123002405
 Ignore-this: b2a7dc163edcab8d9613bfd6907e5166
 
 The Tahoe2PeerSelector returned either NoSharesError or NotEnoughSharesError
 for a variety of error conditions that weren't informatively described by them.
 This patch creates a new error, UploadHappinessError, replaces uses of 
 NoSharesError and NotEnoughSharesError with it, and alters the error message
 raised with the errors to be more in line with the new servers_of_happiness
 behavior. See ticket #834 for more information.
] 
[Eliminate overcounting iof servers_of_happiness in Tahoe2PeerSelector; also reorganize some things.
Kevan Carstensen <kevan@isnotajoke.com>**20091118014542
 Ignore-this: a6cb032cbff74f4f9d4238faebd99868
] 
[Change stray "shares_of_happiness" to "servers_of_happiness"
Kevan Carstensen <kevan@isnotajoke.com>**20091116212459
 Ignore-this: 1c971ba8c3c4d2e7ba9f020577b28b73
] 
[Alter Tahoe2PeerSelector to make sure that it recognizes existing shares on readonly servers, fixing an issue in #778
Kevan Carstensen <kevan@isnotajoke.com>**20091116192805
 Ignore-this: 15289f4d709e03851ed0587b286fd955
] 
[Alter 'immutable/encode.py' and 'immutable/upload.py' to use servers_of_happiness instead of shares_of_happiness.
Kevan Carstensen <kevan@isnotajoke.com>**20091104111222
 Ignore-this: abb3283314820a8bbf9b5d0cbfbb57c8
] 
[Alter the signature of set_shareholders in IEncoder to add a 'servermap' parameter, which gives IEncoders enough information to perform a sane check for servers_of_happiness.
Kevan Carstensen <kevan@isnotajoke.com>**20091104033241
 Ignore-this: b3a6649a8ac66431beca1026a31fed94
] 
[Alter CiphertextDownloader to work with servers_of_happiness
Kevan Carstensen <kevan@isnotajoke.com>**20090924041932
 Ignore-this: e81edccf0308c2d3bedbc4cf217da197
] 
[Revisions of the #778 tests, per reviewers' comments
Kevan Carstensen <kevan@isnotajoke.com>**20100514012542
 Ignore-this: 735bbc7f663dce633caeb3b66a53cf6e
 
 - Fix comments and confusing naming.
 - Add tests for the new error messages suggested by David-Sarah
   and Zooko.
 - Alter existing tests for new error messages.
 - Make sure that the tests continue to work with the trunk.
 - Add a test for a mutual disjointedness assertion that I added to
   upload.servers_of_happiness.
 - Fix the comments to correctly reflect read-onlyness
 - Add a test for an edge case in should_add_server
 - Add an assertion to make sure that share redistribution works as it 
   should
 - Alter tests to work with revised servers_of_happiness semantics
 - Remove tests for should_add_server, since that function no longer exists.
 - Alter tests to know about merge_peers, and to use it before calling 
   servers_of_happiness.
 - Add tests for merge_peers.
 - Add Zooko's puzzles to the tests.
 - Edit encoding tests to expect the new kind of failure message.
 - Edit tests to expect error messages with the word "only" moved as far
   to the right as possible.
 - Extended and cleaned up some helper functions.
 - Changed some tests to call more appropriate helper functions.
 - Added a test for the failing redistribution algorithm
 - Added a test for the progress message
 - Added a test for the upper bound on readonly peer share discovery.
 
] 
[Alter various unit tests to work with the new happy behavior
Kevan Carstensen <kevan@isnotajoke.com>**20100107181325
 Ignore-this: 132032bbf865e63a079f869b663be34a
] 
[Replace "UploadHappinessError" with "UploadUnhappinessError" in tests.
Kevan Carstensen <kevan@isnotajoke.com>**20091205043453
 Ignore-this: 83f4bc50c697d21b5f4e2a4cd91862ca
] 
[Add tests for the behavior described in #834.
Kevan Carstensen <kevan@isnotajoke.com>**20091123012008
 Ignore-this: d8e0aa0f3f7965ce9b5cea843c6d6f9f
] 
[Re-work 'test_upload.py' to be more readable; add more tests for #778
Kevan Carstensen <kevan@isnotajoke.com>**20091116192334
 Ignore-this: 7e8565f92fe51dece5ae28daf442d659
] 
[Test Tahoe2PeerSelector to make sure that it recognizeses existing shares on readonly servers
Kevan Carstensen <kevan@isnotajoke.com>**20091109003735
 Ignore-this: 12f9b4cff5752fca7ed32a6ebcff6446
] 
[Add more tests for comment:53 in ticket #778
Kevan Carstensen <kevan@isnotajoke.com>**20091104112849
 Ignore-this: 3bb2edd299a944cc9586e14d5d83ec8c
] 
[Add a test for upload.shares_by_server
Kevan Carstensen <kevan@isnotajoke.com>**20091104111324
 Ignore-this: f9802e82d6982a93e00f92e0b276f018
] 
[Minor tweak to an existing test -- make the first server read-write, instead of read-only
Kevan Carstensen <kevan@isnotajoke.com>**20091104034232
 Ignore-this: a951a46c93f7f58dd44d93d8623b2aee
] 
[Alter tests to use the new form of set_shareholders
Kevan Carstensen <kevan@isnotajoke.com>**20091104033602
 Ignore-this: 3deac11fc831618d11441317463ef830
] 
[Refactor some behavior into a mixin, and add tests for the behavior described in #778
"Kevan Carstensen" <kevan@isnotajoke.com>**20091030091908
 Ignore-this: a6f9797057ca135579b249af3b2b66ac
] 
[Alter NoNetworkGrid to allow the creation of readonly servers for testing purposes.
Kevan Carstensen <kevan@isnotajoke.com>**20091018013013
 Ignore-this: e12cd7c4ddeb65305c5a7e08df57c754
] 
[Update 'docs/architecture.txt' to reflect readonly share discovery
kevan@isnotajoke.com**20100514003852
 Ignore-this: 7ead71b34df3b1ecfdcfd3cb2882e4f9
] 
[Alter the wording in docs/architecture.txt to more accurately describe the servers_of_happiness behavior.
Kevan Carstensen <kevan@isnotajoke.com>**20100428002455
 Ignore-this: 6eff7fa756858a1c6f73728d989544cc
] 
[Alter wording in 'interfaces.py' to be correct wrt #778
"Kevan Carstensen" <kevan@isnotajoke.com>**20091205034005
 Ignore-this: c9913c700ac14e7a63569458b06980e0
] 
[Update 'docs/configuration.txt' to reflect the servers_of_happiness behavior.
Kevan Carstensen <kevan@isnotajoke.com>**20091205033813
 Ignore-this: 5e1cb171f8239bfb5b565d73c75ac2b8
] 
[Clarify quickstart instructions for installing pywin32
david-sarah@jacaranda.org**20100511180300
 Ignore-this: d4668359673600d2acbc7cd8dd44b93c
] 
[web: add a simple test that you can load directory.xhtml
zooko@zooko.com**20100510063729
 Ignore-this: e49b25fa3c67b3c7a56c8b1ae01bb463
] 
[setup: fix typos in misc/show-tool-versions.py
zooko@zooko.com**20100510063615
 Ignore-this: 2181b1303a0e288e7a9ebd4c4855628
] 
[setup: show code-coverage tool versions in show-tools-versions.py
zooko@zooko.com**20100510062955
 Ignore-this: 4b4c68eb3780b762c8dbbd22b39df7cf
] 
[docs: update README, mv it to README.txt, update setup.py
zooko@zooko.com**20100504094340
 Ignore-this: 40e28ca36c299ea1fd12d3b91e5b421c
] 
[Dependency on Windmill test framework is not needed yet.
david-sarah@jacaranda.org**20100504161043
 Ignore-this: be088712bec650d4ef24766c0026ebc8
] 
[tests: pass z to tar so that BSD tar will know to ungzip
zooko@zooko.com**20100504090628
 Ignore-this: 1339e493f255e8fc0b01b70478f23a09
] 
[setup: update comments and URLs in setup.cfg
zooko@zooko.com**20100504061653
 Ignore-this: f97692807c74bcab56d33100c899f829
] 
[setup: reorder and extend the show-tool-versions script, the better to glean information about our new buildslaves
zooko@zooko.com**20100504045643
 Ignore-this: 836084b56b8d4ee8f1de1f4efb706d36
] 
[CLI: Support for https url in option --node-url
Francois Deppierraz <francois@ctrlaltdel.ch>**20100430185609
 Ignore-this: 1717176b4d27c877e6bc67a944d9bf34
 
 This patch modifies the regular expression used for verifying of '--node-url'
 parameter.  Support for accessing a Tahoe gateway over HTTPS was already
 present, thanks to Python's urllib.
 
] 
[backupdb.did_create_directory: use REPLACE INTO, not INSERT INTO + ignore error
Brian Warner <warner@lothar.com>**20100428050803
 Ignore-this: 1fca7b8f364a21ae413be8767161e32f
 
 This handles the case where we upload a new tahoe directory for a
 previously-processed local directory, possibly creating a new dircap (if the
 metadata had changed). Now we replace the old dirhash->dircap record. The
 previous behavior left the old record in place (with the old dircap and
 timestamps), so we'd never stop creating new directories and never converge
 on a null backup.
] 
["tahoe webopen": add --info flag, to get ?t=info
Brian Warner <warner@lothar.com>**20100424233003
 Ignore-this: 126b0bb6db340fabacb623d295eb45fa
 
 Also fix some trailing whitespace.
] 
[docs: install.html http-equiv refresh to quickstart.html
zooko@zooko.com**20100421165708
 Ignore-this: 52b4b619f9dde5886ae2cd7f1f3b734b
] 
[docs: install.html -> quickstart.html
zooko@zooko.com**20100421155757
 Ignore-this: 6084e203909306bed93efb09d0e6181d
 It is not called "installing" because that implies that it is going to change the configuration of your operating system. It is not called "building" because that implies that you need developer tools like a compiler. Also I added a stern warning against looking at the "InstallDetails" wiki page, which I have renamed to "AdvancedInstall".
] 
[Fix another typo in tahoe_storagespace munin plugin
david-sarah@jacaranda.org**20100416220935
 Ignore-this: ad1f7aa66b554174f91dfb2b7a3ea5f3
] 
[Add dependency on windmill >= 1.3
david-sarah@jacaranda.org**20100416190404
 Ignore-this: 4437a7a464e92d6c9012926b18676211
] 
[licensing: phrase the OpenSSL-exemption in the vocabulary of copyright instead of computer technology, and replicate the exemption from the GPL to the TGPPL
zooko@zooko.com**20100414232521
 Ignore-this: a5494b2f582a295544c6cad3f245e91
] 
[munin-tahoe_storagespace
freestorm77@gmail.com**20100221203626
 Ignore-this: 14d6d6a587afe1f8883152bf2e46b4aa
 
 Plugin configuration rename
 
] 
[setup: add licensing declaration for setuptools (noticed by the FSF compliance folks)
zooko@zooko.com**20100309184415
 Ignore-this: 2dfa7d812d65fec7c72ddbf0de609ccb
] 
[setup: fix error in licensing declaration from Shawn Willden, as noted by the FSF compliance division
zooko@zooko.com**20100309163736
 Ignore-this: c0623d27e469799d86cabf67921a13f8
] 
[CREDITS to Jacob Appelbaum
zooko@zooko.com**20100304015616
 Ignore-this: 70db493abbc23968fcc8db93f386ea54
] 
[desert-island-build-with-proper-versions
jacob@appelbaum.net**20100304013858] 
[docs: a few small edits to try to guide newcomers through the docs
zooko@zooko.com**20100303231902
 Ignore-this: a6aab44f5bf5ad97ea73e6976bc4042d
 These edits were suggested by my watching over Jake Appelbaum's shoulder as he completely ignored/skipped/missed install.html and also as he decided that debian.txt wouldn't help him with basic installation. Then I threw in a few docs edits that have been sitting around in my sandbox asking to be committed for months.
] 
[TAG allmydata-tahoe-1.6.1
david-sarah@jacaranda.org**20100228062314
 Ignore-this: eb5f03ada8ea953ee7780e7fe068539
] 
Patch bundle hash:
a00268ff3d42035ddef8cc426ca34c3ea9537115