Ticket #534: unicode.dpatch

Tue Feb 24 01:06:57 CET 2009  francois@ctrlaltdel.ch
  * CLI.txt: Document behaviour in presence of unicode filename

Tue Feb 24 01:12:51 CET 2009  francois@ctrlaltdel.ch
  * cli.py: Convert filenames and filepaths argument into unicode objects

Tue Feb 24 01:15:54 CET 2009  francois@ctrlaltdel.ch
  * common.py: support unicode alias names

Tue Feb 24 01:16:59 CET 2009  francois@ctrlaltdel.ch
  * tahoe_add_alias.py: support unicode alias names

Tue Feb 24 01:21:18 CET 2009  francois@ctrlaltdel.ch
  * tahoe_ls.py: convert filenames using unicode_to_stdout()

Tue Feb 24 01:22:48 CET 2009  francois@ctrlaltdel.ch
  * tahoe_manifest.py: use unicode_to_stdout() instead of .encode("utf-8") to avoid an exception if terminal doesn't support utf-8

Tue Feb 24 01:23:45 CET 2009  francois@ctrlaltdel.ch
  * tahoe_mkdir.py: path is now unicode and needs conversion before ending up in an url

Tue Feb 24 01:25:13 CET 2009  francois@ctrlaltdel.ch
  * test_cli.py: add a bunch of unicode tests

Tue Feb 24 02:00:21 CET 2009  francois@ctrlaltdel.ch
  * stringutils.py: common functions taking care of encoding conversions

Tue Feb 24 02:02:47 CET 2009  francois@ctrlaltdel.ch
  * tahoe_cp.py: handle unicode filename if the system locale supports it, print an error message otherwise

Tue Feb 24 02:03:08 CET 2009  francois@ctrlaltdel.ch
  * tahoe_backup.py: handle unicode filenames if locale settings supports it, print an error message otherwise

Tue Feb 24 12:45:49 CET 2009  francois@ctrlaltdel.ch
  * tahoe_backup.py: fix conflict

New patches:

[CLI.txt: Document behaviour in presence of unicode filename
francois@ctrlaltdel.ch**20090224000657] hunk ./docs/frontends/CLI.txt 94
-In Tahoe v1.3.0, passing non-ascii characters to the cli is not guaranteed to
-work, although it might work on your platform, especially if your platform
-uses utf-8 encoding.
+Filenames containing non-ascii characters are supported on the commande
+line if your terminal is correctly configured for UTF-8 support. This is
+usually the case on moderns GNU/Linux distributions.
+
+If your terminal doesn't support UTF-8, you will still be able to list
+directories but non-ascii characters will be replaced by a question mark
+(?) on display.
+
+Reading from and writing to files whose name contain accentuated
+characters is also supported when your system correctly understand them.
+Under Unix, this is usually handled by locale settings. If Tahoe cannot
+correctly decode a filename, it will raise an error. In such case,
+you'll need to correct the name of your file, perhaps with help from
+tools such as convmv.

[cli.py: Convert filenames and filepaths argument into unicode objects
francois@ctrlaltdel.ch**20090224001251] {
hunk ./src/allmydata/scripts/cli.py 4
+from allmydata.util.stringutils import argv_to_unicode
hunk ./src/allmydata/scripts/cli.py 53
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 58
-        self.alias = alias
+        self.alias = argv_to_unicode(alias)
hunk ./src/allmydata/scripts/cli.py 68
-        self.alias = alias
+        self.alias = argv_to_unicode(alias)
hunk ./src/allmydata/scripts/cli.py 87
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 98
-        self.from_file = arg1
-        self.to_file = arg2
+        self.from_file = argv_to_unicode(arg1)
+
+        if arg2:
+            self.to_file = argv_to_unicode(arg2)
+        else:
+            self.to_file = None
+
hunk ./src/allmydata/scripts/cli.py 140
-            self.from_file = arg1
-            self.to_file = arg2
+            self.from_file = argv_to_unicode(arg1)
+            self.to_file =  argv_to_unicode(arg2)
hunk ./src/allmydata/scripts/cli.py 143
-            self.from_file = arg1 # might be "-"
+            self.from_file = argv_to_unicode(arg1) # might be "-"
hunk ./src/allmydata/scripts/cli.py 148
-        if self.from_file == "-":
+        if self.from_file == u"-":
hunk ./src/allmydata/scripts/cli.py 182
-        self.sources = args[:-1]
-        self.destination = args[-1]
+        self.sources = map(argv_to_unicode, args[:-1])
+        self.destination = argv_to_unicode(args[-1])
hunk ./src/allmydata/scripts/cli.py 187
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 194
-        self.from_file = frompath
-        self.to_file = topath
+        self.from_file = argv_to_unicode(frompath)
+        self.to_file = argv_to_unicode(topath)
hunk ./src/allmydata/scripts/cli.py 202
-        self.from_file = frompath
-        self.to_file = topath
+        self.from_file = argv_to_unicode(frompath)
+        self.to_file = argv_to_unicode(topath)
hunk ./src/allmydata/scripts/cli.py 227
-        self.from_dir = localdir
-        self.to_dir = topath
+        self.from_dir = argv_to_unicode(localdir)
+        self.to_dir = argv_to_unicode(topath)
hunk ./src/allmydata/scripts/cli.py 276
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 291
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 303
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 318
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/cli.py 334
-        self.where = where
+        self.where = argv_to_unicode(where)
hunk ./src/allmydata/scripts/common.py 3
+import codecs
}

[common.py: support unicode alias names
francois@ctrlaltdel.ch**20090224001554] {
hunk ./src/allmydata/scripts/common.py 5
-
+from allmydata.util.stringutils import unicode_to_url
hunk ./src/allmydata/scripts/common.py 104
-        f = open(aliasfile, "r")
+        f = codecs.open(aliasfile, "r", "utf-8")
hunk ./src/allmydata/scripts/common.py 111
-            cap = cap.strip()
+            cap = cap.strip().encode('ascii')
hunk ./src/allmydata/scripts/common.py 150
-    return "/".join([urllib.quote(s) for s in segments])
+    return "/".join([urllib.quote(unicode_to_url(s)) for s in segments])
}

[tahoe_add_alias.py: support unicode alias names
francois@ctrlaltdel.ch**20090224001659] {
hunk ./src/allmydata/scripts/tahoe_add_alias.py 3
+import codecs
+import sys
hunk ./src/allmydata/scripts/tahoe_add_alias.py 8
+from allmydata.util.stringutils import unicode_to_stdout
hunk ./src/allmydata/scripts/tahoe_add_alias.py 58
-    f = open(aliasfile, "a")
+    f = codecs.open(aliasfile, "a", "utf-8")
hunk ./src/allmydata/scripts/tahoe_add_alias.py 61
-    print >>stdout, "Alias '%s' created" % (alias,)
+    print >>stdout, "Alias '%s' created" % (unicode_to_stdout(alias),)
}

[tahoe_ls.py: convert filenames using unicode_to_stdout()
francois@ctrlaltdel.ch**20090224002118] {
hunk ./src/allmydata/scripts/tahoe_ls.py 6
+from allmydata.util.stringutils import unicode_to_stdout
hunk ./src/allmydata/scripts/tahoe_ls.py 116
-        line.append(name + classify)
+        line.append(unicode_to_stdout(name) + classify)
}

[tahoe_manifest.py: use unicode_to_stdout() instead of .encode("utf-8") to avoid an exception if terminal doesn't support utf-8
francois@ctrlaltdel.ch**20090224002248] hunk ./src/allmydata/scripts/tahoe_manifest.py 71
-                    print >>stdout, d["cap"], "/".join([p.encode("utf-8")
+                    print >>stdout, d["cap"], "/".join([unicode_to_stdout(p)

[tahoe_mkdir.py: path is now unicode and needs conversion before ending up in an url
francois@ctrlaltdel.ch**20090224002345] {
hunk ./src/allmydata/scripts/tahoe_mkdir.py 5
+from allmydata.util.stringutils import unicode_to_url
hunk ./src/allmydata/scripts/tahoe_mkdir.py 35
-                                           urllib.quote(path))
+                                           urllib.quote(unicode_to_url(path)))
}

[test_cli.py: add a bunch of unicode tests
francois@ctrlaltdel.ch**20090224002513] {
hunk ./src/allmydata/test/test_cli.py 3
+import sys
hunk ./src/allmydata/test/test_cli.py 426
+        d.addCallback(lambda res: self.do_cli("create-alias", "études"))
+        def _check_create_unicode((rc,stdout,stderr)):
+            self.failUnlessEqual(rc, 0)
+            self.failIf(stderr)
+
+            # If stdout only supports ascii, accentuated characters are
+            # being replaced by '?'
+            if sys.stdout.encoding == "ANSI_X3.4-1968":
+                self.failUnless("Alias '?tudes' created" in stdout)
+            else:
+                self.failUnless("Alias 'études' created" in stdout)
+
+            aliases = get_aliases(self.get_clientdir())
+            self.failUnless(aliases[u"études"].startswith("URI:DIR2:"))
+        d.addCallback(_check_create_unicode)
+
+        d.addCallback(lambda res: self.do_cli("ls", "études:"))
+        def _check_ls1((rc, stdout, stderr)):
+            self.failUnlessEqual(rc, 0)
+            self.failIf(stderr)
+
+            self.failUnlessEqual(stdout, "")
+        d.addCallback(_check_ls1)
+
+        d.addCallback(lambda res: self.do_cli("put", "-", "études:uploaded.txt",
+          stdin="Blah blah blah"))
+
+        d.addCallback(lambda res: self.do_cli("ls", "études:"))
+        def _check_ls2((rc, stdout, stderr)):
+            self.failUnlessEqual(rc, 0)
+            self.failIf(stderr)
+
+            self.failUnlessEqual(stdout, "uploaded.txt\n")
+        d.addCallback(_check_ls2)
+
hunk ./src/allmydata/test/test_cli.py 685
+    def test_immutable_from_file_unicode(self):
+        # tahoe put file.txt "à trier.txt"
+        self.basedir = os.path.dirname(self.mktemp())
+        self.set_up_grid()
+
+        rel_fn = os.path.join(self.basedir, "DATAFILE")
+        abs_fn = os.path.abspath(rel_fn)
+        # we make the file small enough to fit in a LIT file, for speed
+        DATA = "short file"
+        f = open(rel_fn, "w")
+        f.write(DATA)
+        f.close()
+
+        d = self.do_cli("create-alias", "tahoe")
+
+        d.addCallback(lambda res:
+                      self.do_cli("put", rel_fn, "à trier.txt"))
+        def _uploaded((rc,stdout,stderr)):
+            readcap = stdout.strip()
+            self.failUnless(readcap.startswith("URI:LIT:"))
+            self.failUnless("201 Created" in stderr, stderr)
+            self.readcap = readcap
+        d.addCallback(_uploaded)
+
+        d.addCallback(lambda res:
+                      self.do_cli("get", "tahoe:à trier.txt"))
+        d.addCallback(lambda (rc,stdout,stderr):
+                      self.failUnlessEqual(stdout, DATA))
+
+        return d
+
hunk ./src/allmydata/test/test_cli.py 726
+        d = self.do_cli("create-alias", "tahoe")
+
+        # Use unicode strings when calling os functions
+        if sys.getfilesystemencoding() == "ANSI_X3.4-1968":
+            fn1 = os.path.join(self.basedir, u"Artonwall")
+        else:
+            fn1 = os.path.join(self.basedir, u"Ärtonwall")
hunk ./src/allmydata/test/test_cli.py 734
-        fn1 = os.path.join(self.basedir, "Ärtonwall")
hunk ./src/allmydata/test/test_cli.py 736
+        d.addCallback(lambda res: self.do_cli("cp", fn1.encode('utf-8'), "tahoe:Ärtonwall"))
+
+        d.addCallback(lambda res: self.do_cli("get", "tahoe:Ärtonwall"))
+        d.addCallback(lambda (rc,out,err): self.failUnlessEqual(out, DATA1))
hunk ./src/allmydata/test/test_cli.py 741
-        fn2 = os.path.join(self.basedir, "Metallica")
+
+        fn2 = os.path.join(self.basedir, u"Metallica")
hunk ./src/allmydata/test/test_cli.py 749
-        d = self.do_cli("create-alias", "tahoe")
-        d.addCallback(lambda res: self.do_cli("cp", fn1, "tahoe:"))
-        d.addCallback(lambda res: self.do_cli("cp", fn2, "tahoe:"))
-
-        d.addCallback(lambda res: self.do_cli("get", "tahoe:Ärtonwall"))
-        d.addCallback(lambda (rc,out,err): self.failUnlessEqual(out, DATA1))
+        d.addCallback(lambda res: self.do_cli("cp", fn2.encode('utf-8'), "tahoe:"))
hunk ./src/allmydata/test/test_cli.py 754
+        d.addCallback(lambda res: self.do_cli("ls", "tahoe:"))
+
hunk ./src/allmydata/test/test_cli.py 757
-    test_unicode_filename.todo = "This behavior is not yet supported, although it does happen to work (for reasons that are ill-understood) on many platforms.  See issue ticket #534."
hunk ./src/allmydata/test/test_cli.py 775
+class Mkdir(GridTestMixin, CLITestMixin, unittest.TestCase):
+    def test_unicode_mkdir(self):
+        self.basedir = os.path.dirname(self.mktemp())
+        self.set_up_grid()
+
+        d = self.do_cli("create-alias", "tahoe")
+        d.addCallback(lambda res: self.do_cli("mkdir", "tahoe:Motörhead"))
+
+        return d
+ 
+
hunk ./src/allmydata/test/test_cli.py 820
+        if sys.getfilesystemencoding() == "ANSI_X3.4-1968":
+            self.writeto(u"parent/artonwall.txt", "Marmelade Jacuzzi")
+        else:
+            self.writeto(u"parent/ärtonwall.txt", "Marmelade Jacuzzi")
+
hunk ./src/allmydata/test/test_cli.py 849
-            # foo.txt, bar.txt, blah.txt
-            self.failUnlessEqual(fu, 3)
+            # foo.txt, bar.txt, blah.txt, ärtonwall.txt
+            self.failUnlessEqual(fu, 4)
hunk ./src/allmydata/test/test_cli.py 899
-                # foo.txt, bar.txt, blah.txt
+                # foo.txt, bar.txt, blah.txt, ärtonwall.txt
hunk ./src/allmydata/test/test_cli.py 901
-                self.failUnlessEqual(fr, 3)
+                self.failUnlessEqual(fr, 4)
hunk ./src/allmydata/test/test_cli.py 929
-                self.failUnlessEqual(fchecked, 3)
+                self.failUnlessEqual(fchecked, 4)
hunk ./src/allmydata/test/test_cli.py 931
-                self.failUnlessEqual(fr, 3)
+                self.failUnlessEqual(fr, 4)
hunk ./src/allmydata/test/test_cli.py 977
-                # old bar.txt
-                self.failUnlessEqual(fr, 1)
+                # old bar.txt, ärtonwall.txt
+                self.failUnlessEqual(fr, 2)
hunk ./src/allmydata/test/test_cli.py 1017
-            self.failUnlessEqual(fu, 5)
+            self.failUnlessEqual(fu, 6)
}

[stringutils.py: common functions taking care of encoding conversions
francois@ctrlaltdel.ch**20090224010021] {
addfile ./src/allmydata/util/stringutils.py
hunk ./src/allmydata/util/stringutils.py 1
+"""
+Functions used to convert inputs from whatever encoding used in the system to
+unicode and back.
+
+TODO:
+  * Accept two cli arguments --argv-encoding and --filesystem-encoding
+"""
+
+import sys
+from allmydata.util.assertutil import precondition
+from twisted.python import usage
+
+def argv_to_unicode(s):
+    """
+    Decode given argv element to unicode.
+    """
+    # sys.argv encoding detection in Python is not trivial so utf-8 is
+    # currently used by default and an informative error message is given if
+    # the argument cannot be correctly decoded.
+
+    precondition(isinstance(s, str), s)
+    try:
+        return unicode(s, 'utf-8')
+    except UnicodeEncodeError:
+        raise usageError("Argument '%s' cannot be decoded as UTF-8." % s)
+
+def fs_to_unicode(s):
+    """
+    Decode a filename (or a directory name) to unicode using the same encoding
+    as the filesystem.
+    """
+    # Filename encoding detection is a little bit better thanks to
+    # getfilesystemencoding() in the sys module. However, filenames can be
+    # encoded using another encoding than the one used on the filesystem.
+    
+    precondition(isinstance(s, str), s)
+    encoding = sys.getfilesystemencoding()
+    try:
+        return unicode(s, encoding)
+    except UnicodeDecodeError:
+        raise usage.UsageError("Filename '%s' cannot be decoded using the current encoding of your filesystem (%s). Please rename this file." % (s, encoding))
+
+def unicode_to_fs(s):
+    """
+    Encode an unicode object used in file or directoy name.
+    """
+
+    precondition(isinstance(s, unicode), s)
+    encoding = sys.getfilesystemencoding()
+    try:
+        return s.encode(encoding)
+    except UnicodeEncodeError:
+        raise usage.UsageError("Filename '%s' cannot be encoded using the current encoding of your filesystem (%s). Please configure your locale correctly or rename this file." % (s, encoding))
+
+def unicode_to_url(s):
+    """
+    Encode an unicode object used in an URL.
+    """
+    # According to RFC 2718, non-ascii characters in url's must be UTF-8 encoded.
+
+    precondition(isinstance(s, unicode), s)
+    return s.encode('utf-8')
+
+def unicode_to_stdout(s):
+    """
+    Encode an unicode object for representation on stdout.
+    """
+
+    precondition(isinstance(s, unicode), s)
+    return s.encode(sys.stdout.encoding, 'replace')
}

[tahoe_cp.py: handle unicode filename if the system locale supports it, print an error message otherwise
francois@ctrlaltdel.ch**20090224010247] {
hunk ./src/allmydata/scripts/tahoe_cp.py 5
+import sys
hunk ./src/allmydata/scripts/tahoe_cp.py 9
+from twisted.python import usage
+from allmydata.util.stringutils import fs_to_unicode, unicode_to_fs, unicode_to_url
+from allmydata.util.assertutil import precondition
hunk ./src/allmydata/scripts/tahoe_cp.py 72
+        precondition(isinstance(pathname, unicode), pathname)
hunk ./src/allmydata/scripts/tahoe_cp.py 83
+        precondition(isinstance(pathname, unicode), pathname)
hunk ./src/allmydata/scripts/tahoe_cp.py 96
+        precondition(isinstance(pathname, unicode), pathname)
hunk ./src/allmydata/scripts/tahoe_cp.py 110
+        precondition(isinstance(pathname, unicode), pathname)
+
hunk ./src/allmydata/scripts/tahoe_cp.py 120
-        children = os.listdir(self.pathname)
+        children = os.listdir(unicode_to_fs(self.pathname))
hunk ./src/allmydata/scripts/tahoe_cp.py 122
+            n = fs_to_unicode(n)
hunk ./src/allmydata/scripts/tahoe_cp.py 138
+        precondition(isinstance(pathname, unicode), pathname)
+
hunk ./src/allmydata/scripts/tahoe_cp.py 148
-        children = os.listdir(self.pathname)
+        children = os.listdir(unicode_to_fs(self.pathname))
hunk ./src/allmydata/scripts/tahoe_cp.py 150
+            n = fs_to_unicode(n)
hunk ./src/allmydata/scripts/tahoe_cp.py 172
+        precondition(isinstance(name, unicode), name)
hunk ./src/allmydata/scripts/tahoe_cp.py 174
-        outf = open(pathname, "wb")
+        outf = open(unicode_to_fs(pathname), "wb")
hunk ./src/allmydata/scripts/tahoe_cp.py 359
-                                                   urllib.quote(name.encode('utf-8'))])
+                                                   urllib.quote(unicode_to_url(name))])
}

[tahoe_backup.py: handle unicode filenames if locale settings supports it, print an error message otherwise
francois@ctrlaltdel.ch**20090224010308] {
hunk ./src/allmydata/scripts/tahoe_backup.py 6
+import sys
hunk ./src/allmydata/scripts/tahoe_backup.py 12
+from allmydata.util.stringutils import fs_to_unicode, unicode_to_fs, unicode_to_stdout
+from allmydata.util.assertutil import precondition
+from twisted.python import usage
hunk ./src/allmydata/scripts/tahoe_backup.py 236
-            print >>self.options.stdout, msg
+            print >>self.options.stdout, unicode_to_stdout(msg)
hunk ./src/allmydata/scripts/tahoe_backup.py 239
+        precondition(isinstance(localpath, unicode), localpath)
merger 0.0 (
hunk ./src/allmydata/scripts/tahoe_backup.py 248
-        for child in os.listdir(localpath):
+        for child in self.options.filter_listdir(os.listdir(localpath)):
hunk ./src/allmydata/scripts/tahoe_backup.py 248
-        for child in os.listdir(localpath):
-            childpath = os.path.join(localpath, child)
+        for child in os.listdir(unicode_to_fs(localpath)):
+            child = fs_to_unicode(child)
+            childpath = os.path.join(localpath, child)
)
hunk ./src/allmydata/scripts/tahoe_backup.py 334
+        precondition(isinstance(childpath, unicode), childpath)
+
hunk ./src/allmydata/scripts/tahoe_backup.py 344
-            infileobj = open(os.path.expanduser(childpath), "rb")
+            infileobj = open(unicode_to_fs(os.path.expanduser(childpath)), "rb")
}

[tahoe_backup.py: fix conflict
francois@ctrlaltdel.ch**20090224114549] hunk ./src/allmydata/scripts/tahoe_backup.py 248
-        for child in os.listdir(localpath):
+        for child in self.options.filter_listdir(os.listdir(unicode_to_fs(localpath))):
+            child = fs_to_unicode(child)

Context:

[test_repairer: change to use faster no_network.GridTestMixin, split Verifier tests into separate cases, refactor judgement funcs into shared methods
warner@lothar.com**20090224041506
 Ignore-this: 584ce72d6276da5edc00562793d4ee53
] 
[immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check
warner@lothar.com**20090224041405
 Ignore-this: b667e8d3192116293babcacdeed42898
 instead of examining the value returned by f.trap, because the latter appears
 to squash exception types down into their base classes (i.e. since
 ShareVersionIncompatible is a subclass of LayoutInvalid,
 f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid).
 
 All this resulted in 'incompatible' shares being misclassified as 'corrupt'.
] 
[immutable/layout.py: wrap to 80 cols, no functional changes
warner@lothar.com**20090224005837
 Ignore-this: 40019480180ec34141506a28d7711608
] 
[test_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a side-effect, fix what I think was a bug: some of the assert-minimal-effort-expended checks were mixing write counts and allocate counts
warner@lothar.com**20090223234227
 Ignore-this: d58bd0a909f9939775730cda4a858cae
] 
[test/no_network.py: add a basic stats provider
warner@lothar.com**20090223233937
 Ignore-this: c9f3cc4eed99cfc36f68938ceff4162c
] 
[tests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted-8.2.0
warner@lothar.com**20090223204312
 Ignore-this: 24c6592141cf64103530c024f93a5b88
] 
[test_checker: improve test coverage for checker results
warner@lothar.com**20090223201943
 Ignore-this: 83e173602f0f4c811a7a9893d85385df
] 
[Fixed tests again so they will pass on windows.
Alberto Berti <alberto@metapensiero.it>**20090223003502
 Ignore-this: 80d5074e7153642a2fa2a77958bfb50d
] 
[misc/*: remove RuntimeError too
warner@lothar.com**20090222233401
 Ignore-this: b76f8a184f75bb28eb9d8002f957936a
] 
[scripts: stop using RuntimeError, for #639
warner@lothar.com**20090222233106
 Ignore-this: 686a424442670fffbd4d1816c284a601
] 
[mutable/publish: stop using RuntimeError, for #639
warner@lothar.com**20090222233056
 Ignore-this: 2a80a661c7850d97357caddad48c6e9d
] 
[remove more RuntimeError from unit tests, for #639
warner@lothar.com**20090222232855
 Ignore-this: 1a1c3e1457f3f29ba7101fe406ee5f43
] 
[stop using RuntimeError in unit tests, for #639
warner@lothar.com**20090222232722
 Ignore-this: 475ce0c0dcd7a1f5ed83ef460312efea
] 
[ftpd/sftpd: stop using RuntimeError, for #639
warner@lothar.com**20090222232426
 Ignore-this: 97001362c4ba9e94b2e254e229b79987
] 
[Added tests for the cse when listdir is an iterator
Alberto Berti <alberto@metapensiero.it>**20090222224356
 Ignore-this: 218fb2aba02c28b4b1e5324bdb5adeaa
] 
[Fixed tests so that they pass also on buildbots.
Alberto Berti <alberto@metapensiero.it>**20090222224311
 Ignore-this: fcb91cd6acf028382411d23d380a4576
] 
[Use failUnlessEqual instead of failUnless(a == b)
Alberto Berti <alberto@metapensiero.it>**20090222224214
 Ignore-this: 8f9144632e3ac9acb4726fb48a083bf4
] 
[Better implementation of filtering algorithm.
Alberto Berti <alberto@metapensiero.it>**20090222224049
 Ignore-this: 67a8bd2f99bcc87ca2443bef13370a87
] 
[Removed '.hgrags' from vcs excludes
Alberto Berti <alberto@metapensiero.it>**20090222223946
 Ignore-this: 3e94c22fc9d85f380ee11fb8bdb4d1e9
] 
[docs: CREDITS to Alberto Berti
zooko@zooko.com**20090222193314
 Ignore-this: 74d370ada3234cce9e58aec15d739f71
] 
[Added tests for the --exclude* options of backup command.
Alberto Berti <alberto@metapensiero.it>**20090222165106
 Ignore-this: f1b931cf2e7929ce47b737c022bca707
] 
[Added --exclude, --exclude-from and --exclude-vcs options to backup command.
Alberto Berti <alberto@metapensiero.it>**20090222170829
 Ignore-this: 4912890229cd54a2f61f14f06bc4afcc
 
 It is still impossible to specify absolute exclusion path, only
 relative. I must check with tar or rsync how they allow them to be
 specified.
] 
[Raise a more explanatory exception for errors encountered during backup processing.
Alberto Berti <alberto@metapensiero.it>**20090222170252
 Ignore-this: f6b8ffe2a903ba07a2c1c59130dac1e4
] 
[Added tests for the fixed alias related command's synopsis
Alberto Berti <alberto@metapensiero.it>**20090222163732
 Ignore-this: 4432b4e88e990ba53a5b3fe0f12db2ac
] 
[Add missing synopsis and descriptions for alias commands.
Alberto Berti <alberto@metapensiero.it>**20090221003106
 Ignore-this: 8aedd03d36d92d912102c7f29e4ca697
] 
[docs: move many specification-like documents into specifications/
warner@lothar.com**20090222054054
 Ignore-this: a4110cc478198c0611205aba1ccf54f4
] 
[test_web.py: increase test coverage of web.status.plural()
warner@lothar.com**20090222000116
 Ignore-this: 3138c9d5d2410d8e1121e9b2ed694169
] 
[crawler: fix performance problems: only save state once per timeslice (not after every bucket), don't start the crawler until 5 minutes after node startup
warner@lothar.com**20090221205649
 Ignore-this: e6551569982bd31d19779ff15c2d6f58
] 
[test_system: oops, don't assume that all files in storage/ are in a deep storage/shares/prefix/si/shnum path, since now the crawler pickle has a short path
warner@lothar.com**20090221061710
 Ignore-this: fde76d0e5cae853014d1bb18b5f17dae
] 
[crawler: tolerate low-resolution system clocks (i.e. windows)
warner@lothar.com**20090221061533
 Ignore-this: 57286a3abcaf44f6d1a78c3c1ad547a5
] 
[BucketCountingCrawler: store just the count, not cycle+count, since it's too easy to make usage mistakes otherwise
warner@lothar.com**20090221035831
 Ignore-this: 573b6f651af74380cdd64059fbbdda4b
] 
[test_storage: startService the server, as is now the standard practice
warner@lothar.com**20090221035755
 Ignore-this: 3999889bd628fe4039bbcf1b29160453
] 
[crawler: load state from the pickle in init, rather than waiting until startService, so get_state() can be called early
warner@lothar.com**20090221035720
 Ignore-this: ecd128a5f4364c0daf4b72d791340b66
] 
[BucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share', because the former is more accurate
warner@lothar.com**20090221034606
 Ignore-this: cf819f63fac9506c878d6c9715ce35b7
] 
[storage: also report space-free-for-root and space-free-for-nonroot, since that helps users understand the space-left-for-tahoe number better
warner@lothar.com**20090221032856
 Ignore-this: 9fdf0475f758acd98b73026677170b45
] 
[storage: add bucket-counting share crawler, add its output (number of files+directories maintained by a storage server) and status to the webapi /storage page
warner@lothar.com**20090221030408
 Ignore-this: 28761c5e076648026bc5f518506db65c
] 
[storage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py
warner@lothar.com**20090221030309
 Ignore-this: 645056428ab797f0b542831c82bf192a
] 
[crawler: add get_progress, clean up get_state
warner@lothar.com**20090221002743
 Ignore-this: 9bea69f154c75b31a53425a8ea67789b
] 
[web/storage: make sure we can handle platforms without os.statvfs too
warner@lothar.com**20090220220353
 Ignore-this: 79d4cb8482a8543b9759dc949c86c587
] 
[crawler: provide for one-shot crawlers, which stop after their first full cycle, for share-upgraders and database-populaters
warner@lothar.com**20090220211911
 Ignore-this: fcdf72c5ffcafa374d376388be6fa5c5
] 
[web: add Storage status page, improve tests
warner@lothar.com**20090220202926
 Ignore-this: e34d5270dcf0237fe72f573f717c7a4
] 
[storage: include reserved_space in stats
warner@lothar.com**20090220202920
 Ignore-this: b5b480fe0abad0148ecad0c1fb47ecae
] 
[web/check_results: sort share identifiers in the sharemap display
warner@lothar.com**20090220182922
 Ignore-this: 5c7bfcee3e15c7082c3653eb8a460960
] 
[webapi: pass client through constructor arguments, remove IClient, should make it easier to test web renderers in isolation
warner@lothar.com**20090220181554
 Ignore-this: e7848cd1bee8faf2ce7aaf040b9bf8e3
] 
[test/no_network: do startService on the storage servers, make it easier to customize the storage servers
warner@lothar.com**20090220022254
 Ignore-this: e62f328721c007e4c5ee023a6efdf66d
] 
[crawler: modify API to support upcoming bucket-counting crawler
warner@lothar.com**20090220013142
 Ignore-this: 808f8382837b13082f8b245db2ebee06
] 
[test_backupdb: make the not-a-database file larger, since the older sqlite-2.3.2 on OS-X is easily fooled
warner@lothar.com**20090220000409
 Ignore-this: 694d2ca5053bb96e91670765d0cedf2e
] 
[web/reliability: add parameter descriptions, adapted from a patch from Terrell Russell.
warner@lothar.com**20090219222918
 Ignore-this: 835f5ab01e1aff31b2ff9febb9a51f3
] 
[test_crawler: hush pyflakes
warner@lothar.com**20090219202340
 Ignore-this: 765d22c9c9682cc86c5205dc130500af
] 
[test_crawler: disable the percentage-of-cpu-used test, since it is too unreliable on our slow buildslaves. But leave the code in place for developers to run by hand.
warner@lothar.com**20090219201654
 Ignore-this: ff7cf5cfa79c6f2ef0cf959495dd989a
] 
[reliability.py: fix the numpy conversion, it was completely broken. Thanks to Terrell Russell for the help.
warner@lothar.com**20090219195515
 Ignore-this: f2b1eb65855111b338e1487feee1bbcf
] 
[reliability: switch to NumPy, since Numeric is deprecated
warner@lothar.com**20090219074435
 Ignore-this: f588a68e9bcd3b0bc3653570882b6fd5
] 
[setup.py: fix pyflakes complaints
warner@lothar.com**20090219073643
 Ignore-this: a314e5456b0a796bc9f70232a119ec68
] 
[move show-tool-versions out of setup.py and into a separate script in misc/ , since setuptools is trying to build and install a bunch of stuff first
warner@lothar.com**20090219073558
 Ignore-this: 9e56bc43026379212e6b6671ed6a1fd4
] 
[test_crawler: don't require >=1 cycle on cygwin
warner@lothar.com**20090219065818
 Ignore-this: b8d2d40f26aeb30a7622479840a04635
] 
[setup.py: add show_tool_versions command, for the benefit of a new buildbot step
warner@lothar.com**20090219062436
 Ignore-this: 21d761c76a033e481831584bedc60c86
] 
[setup.py: wrap to 80 cols, no functional changes
warner@lothar.com**20090219055751
 Ignore-this: d29e57c6ee555f2ee435667b7e13e60b
] 
[crawler: use fileutil.move_info_place in preference to our own version
warner@lothar.com**20090219051342
 Ignore-this: ee4e46f3de965610503ba36b28184db9
] 
[fileutil: add move_into_place(), to perform the standard unix trick of atomically replacing a file, with a fallback for windows
warner@lothar.com**20090219051310
 Ignore-this: c1d35e8ca88fcb223ea194513611c511
] 
[crawler: fix problems on windows and our slow cygwin slave
warner@lothar.com**20090219042431
 Ignore-this: 8019cb0da79ba00c536183a6f57b4cab
] 
[#633: first version of a rate-limited interruptable share-crawler
warner@lothar.com**20090219034633
 Ignore-this: 5d2d30c743e3b096a8e775d5a9b33601
] 
[change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away
warner@lothar.com**20090218222301
 Ignore-this: 740d582f20c93bebf60e21d9a446d3d2
] 
[test_system: split off checker tests to test_deepcheck.py, this file is too big
warner@lothar.com**20090218214234
 Ignore-this: 82bf8db81dfbc98224bbf694054a8761
] 
[break storage.py into smaller pieces in storage/*.py . No behavioral changes.
warner@lothar.com**20090218204655
 Ignore-this: 312d408d1cacc5a764d791b53ebf8f91
] 
[immutable/layout: minor change to repr name
warner@lothar.com**20090218204648
 Ignore-this: c8781ef15b7dea63b39236a1899b86ce
] 
[docs: add lease-tradeoffs diagram
warner@lothar.com**20090218204137
 Ignore-this: c22a589ad465dac846da834c30dc4083
] 
[interfaces.py: allow add/renew/cancel-lease to return Any, so that 1.3.1 clients (the first to use these calls) can tolerate future storage servers which might return something other than None
warner@lothar.com**20090218192903
 Ignore-this: dcbb704a05416ecc66d90fb486c3d75b
] 
[docs/debian.txt: minor edit
warner@lothar.com**20090218032212
 Ignore-this: 64ff1fb163ffca4bcfd920254f1cf866
] 
[add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi.
warner@lothar.com**20090218013243
 Ignore-this: 176b2006cef5041adcb592ee83e084dd
] 
[change RIStorageServer.remote_add_lease to exit silently in case of no-such-bucket, instead of raising IndexError, because that makes the upcoming --add-lease feature faster and less noisy
warner@lothar.com**20090218013053
 Ignore-this: 6fdfcea2c832178f1ce72ab0ff510f3a
] 
[CLI #590: convert 'tahoe deep-check' to streaming form, improve display, add tests
warner@lothar.com**20090217231511
 Ignore-this: 6d88eb94b1c877eacc8c5ca7d0aac776
] 
[interfaces.py: document behavior of add_lease/renew_lease/cancel_lease, before I change it
warner@lothar.com**20090217194809
 Ignore-this: 703c6712926b8edb19d55d790b65a400
] 
[test_backupdb: improve error messages if the test fails
warner@lothar.com**20090217170838
 Ignore-this: ef657e87c66e4304d3e0aca9831b84c
] 
[webapi #590: add streaming deep-check. Still need a CLI tool to use it.
warner@lothar.com**20090217053553
 Ignore-this: a0edd3d2a531c48a64d8397f7e4b208c
] 
[test_web.Grid: change the CHECK() function to make it easier to test t= values with hyphens in them
warner@lothar.com**20090217050034
 Ignore-this: 410c08735347c2057df52f6716520228
] 
[test_web: improve checker-results coverage with a no-network -based test, enhance no-network harness to assist, fix some bugs in web/check_results.py that were exposed
warner@lothar.com**20090217041242
 Ignore-this: fe54bb66a9ae073c002a7af51cd1e18
] 
[web: fix handling of reliability page when Numeric is not available
warner@lothar.com**20090217015658
 Ignore-this: 9d329182f1b2e5f812e5e7eb5f4cf2ed
] 
[test/no_network: update comments with setup timing: no_network takes 50ms, SystemTestMixin takes 2s (on my laptop)
warner@lothar.com**20090217000643
 Ignore-this: cc778fa3219775b25057bfc9491f8f34
] 
[test_upload: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary
warner@lothar.com**20090216234457
 Ignore-this: 80a341d5aa3036d24de98e267499d70d
] 
[test_download: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary
warner@lothar.com**20090216233658
 Ignore-this: ec2febafd2403830519120fb3f3ca04e
] 
[test_dirnode.py: convert Deleter to new no-network gridtest
warner@lothar.com**20090216232348
 Ignore-this: 8041739442ec4db726675e48f9775ae9
] 
[test_cli.py: modify to use the new 'no-network' gridtest instead of SystemTestMixin, which speeds it up from 73s to 43s on my system
warner@lothar.com**20090216232005
 Ignore-this: ec6d010c9182aa72049d1fb894cf890e
] 
[tests: fix no_network framework to work with upload/download and checker
warner@lothar.com**20090216231947
 Ignore-this: 74b4dbd66b8384ae7c7544969fe4f744
] 
[client.py: improve docstring
warner@lothar.com**20090216231532
 Ignore-this: bbaa9e3f63fdb0048e3125c4681b2d1f
] 
[test_cli: add test coverage for help strings
warner@lothar.com**20090216210833
 Ignore-this: d2020849107f687448e159a19d0e5dab
] 
[test/no_network: new test harness, like system-test but doesn't use the network so it's faster
warner@lothar.com**20090216205844
 Ignore-this: 31678f7bdef30b0216fd657fc6145534
] 
[interfaces.py: minor docstring edit
warner@lothar.com**20090216205816
 Ignore-this: cec3855070197f7920b370f95e8b07bd
] 
[setup: if you sdist_dsc (to produce the input files for dpkg-buildpackage) then run darcsver first
zooko@zooko.com**20090216201558
 Ignore-this: b85be51b3d4a9a19a3366e690f1063e2
] 
[doc: a few edits to docs made after the 1.3.0 release
zooko@zooko.com**20090216201539
 Ignore-this: dbff3b929d88134d862f1dffd1ef068a
] 
[test_cli: improve test coverage slightly
warner@lothar.com**20090216030451
 Ignore-this: e01ccc6a6fb44aaa4fb14fe8669e2065
] 
[test_util: get almost full test coverage of dictutil, starting with the original pyutil tests as a base. The remaining three uncovered lines involve funny cases of ValueOrderedDict that I can't figure out how to get at
warner@lothar.com**20090216023210
 Ignore-this: dc1f0c6d8c003c0ade38bc8f8516b04d
] 
[provisioning/reliability: add tests, hush pyflakes, remove dead code, fix web links
warner@lothar.com**20090215222451
 Ignore-this: 7854df3e0130d9388f06efd4c797262f
] 
[util/statistics: add tests, fix mean_repair_cost
warner@lothar.com**20090215222326
 Ignore-this: c576eabc74c23b170702018fc3c122d9
] 
[test_repairer: hush pyflakes
warner@lothar.com**20090215222310
 Ignore-this: 875eb52e86077cda77efd02da77f8cfa
] 
[lossmodel.lyx: move draft paper into docs/proposed/, since it's unfinished
warner@lothar.com**20090215221905
 Ignore-this: 7f7ee204e47fd66932759c94deefe68
] 
[build a 'reliability' web page, with a simulation of file decay and repair over time
warner@lothar.com**20090213234234
 Ignore-this: 9e9623eaac7b0637bbd0071f082bd345
] 
[More lossmodel work, on repair.
Shawn Willden <shawn-tahoe@willden.org>**20090116025648] 
[Loss model work (temp1)
Shawn Willden <shawn@willden.org>**20090115030058] 
[Statistics module
Shawn Willden <shawn-tahoe@willden.org>**20090114021235
 
 Added a statistics module for calculating various facets of
 share survival statistics.
] 
[docs: relnotes-short.txt
zooko@zooko.com**20090215163510
 Ignore-this: 683649bb13499bbe0e5cea2e1716ff59
 linkedin.com imposed a strict limit on the number of characters I could post.  This forced me to prune and prune and edit and edit until relnotes.txt was a quarter of its former size.  Here's the short version.
] 
[TAG allmydata-tahoe-1.3.0
zooko@zooko.com**20090214000556
 Ignore-this: aa6c9a31a14a58ad2298cb7b08d3ea70
] 
Patch bundle hash:
f887deca1f988b992a6399f6b38d155b5ce00bfd