Ticket #1555: check-miscaptures-v3.darcs.patch

6 patches for repository /home/davidsarah/tahoe/1.9alpha2:

Fri Oct  7 08:41:21 BST 2011  david-sarah@jacaranda.org
  * Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) This is a rewritten version that reports much fewer false positives, by determining captured variables more accurately. fixes #1555

Sun Oct  9 05:40:22 BST 2011  david-sarah@jacaranda.org
  * check-miscaptures.py: check while loops and list comprehensions as well as for loops. Also fix a pyflakes warning. refs #1555

Sun Oct  9 05:47:10 BST 2011  david-sarah@jacaranda.org
  * check-miscaptures.py: handle destructuring function arguments correctly. refs #1555

Sun Oct  9 05:48:00 BST 2011  david-sarah@jacaranda.org
  * check-miscaptures.py: Python doesn't really have declarations; report the topmost assignment. refs #1555

Sun Oct  9 05:50:23 BST 2011  david-sarah@jacaranda.org
  * check-miscaptures.py: handle corner cases around default arguments correctly. Also make a minor optimization when there are no assigned variables to consider. refs #1555

Sun Oct  9 06:03:01 BST 2011  david-sarah@jacaranda.org
  * check-miscaptures.py: report the number of files that were not analysed due to syntax errors (and don't count them in the number of suspicious captures). refs #1555

New patches:

[Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) This is a rewritten version that reports much fewer false positives, by determining captured variables more accurately. fixes #1555
david-sarah@jacaranda.org**20111007074121
 Ignore-this: 51318e9678d132c374ea557ab955e79e
] {
hunk ./Makefile 124
        false
 endif
 
-code-checks: build version-and-path check-interfaces -find-trailing-spaces -check-umids pyflakes
+code-checks: build version-and-path check-interfaces check-miscaptures -find-trailing-spaces -check-umids pyflakes
 
 version-and-path:
        $(TAHOE) --version-and-path
hunk ./Makefile 133
        $(TAHOE) @misc/coding_tools/check-interfaces.py 2>&1 |tee violations.txt
        @echo
 
+check-miscaptures:
+       $(PYTHON) misc/coding_tools/check-miscaptures.py $(SOURCES) 2>&1 |tee miscaptures.txt
+       @echo
+
 pyflakes:
        $(PYTHON) -OOu `which pyflakes` $(SOURCES) |sort |uniq
        @echo
addfile ./misc/coding_tools/check-miscaptures.py
hunk ./misc/coding_tools/check-miscaptures.py 1
+#! /usr/bin/python
+
+import os, sys, compiler, traceback
+from compiler.ast import Node, For, AssName, Name, Lambda, Function
+
+
+def check_source(source):
+    return check_thing(compiler.parse, source)
+
+def check_file(path):
+    return check_thing(compiler.parseFile, path)
+
+def check_thing(parser, thing):
+    try:
+        ast = parser(thing)
+    except SyntaxError, e:
+        return [e]
+    else:
+        results = []
+        check_ast(ast, results)
+        return results
+
+def check_ast(ast, results):
+    """Check a node outside a 'for' loop."""
+    if isinstance(ast, For):
+        check_for(ast, results)
+    else:
+        for child in ast.getChildNodes():
+            if isinstance(ast, Node):
+                check_ast(child, results)
+
+def check_for(ast, results):
+    """Check a particular outer 'for' loop."""
+
+    declared = {}  # maps name to lineno of declaration
+    nested = set()
+    collect_declared_and_nested(ast, declared, nested)
+
+    # For each nested function...
+    for funcnode in nested:
+        # Check for captured variables in this function.
+        captured = set()
+        collect_captured(funcnode, declared, captured)
+        for name in captured:
+            # We want to report the outermost capturing function
+            # (since that is where the workaround will need to be
+            # added), and the variable declaration. Just one report
+            # per capturing function per variable will do.
+            results.append(make_result(funcnode, name, declared[name]))
+
+        # Check each node in the function body in case it
+        # contains another 'for' loop.
+        childnodes = funcnode.getChildNodes()[len(funcnode.defaults):]
+        for child in childnodes:
+            check_ast(funcnode, results)
+
+def collect_declared_and_nested(ast, declared, nested):
+    """
+    Collect the names declared in this 'for' loop, not including
+    names declared in nested functions. Also collect the nodes of
+    functions that are nested one level deep.
+    """
+    if isinstance(ast, AssName):
+        declared[ast.name] = ast.lineno
+    else:
+        childnodes = ast.getChildNodes()
+        if isinstance(ast, (Lambda, Function)):
+            nested.add(ast)
+
+            # The default argument expressions are "outside" the
+            # function, even though they are children of the
+            # Lambda or Function node.
+            childnodes = childnodes[:len(ast.defaults)]
+
+        for child in childnodes:
+            if isinstance(ast, Node):
+                collect_declared_and_nested(child, declared, nested)
+
+def collect_captured(ast, declared, captured):
+    """Collect any captured variables that are also in declared."""
+    if isinstance(ast, Name):
+        if ast.name in declared:
+            captured.add(ast.name)
+    else:
+        childnodes = ast.getChildNodes()
+
+        if isinstance(ast, (Lambda, Function)):
+            # Formal parameters of the function are excluded from
+            # captures we care about in subnodes of the function body.
+            declared = declared.copy()
+            for argname in ast.argnames:
+                if argname in declared:
+                    del declared[argname]
+
+            for child in childnodes[len(ast.defaults):]:
+                collect_captured(child, declared, captured)
+
+            # The default argument expressions are "outside" the
+            # function, even though they are children of the
+            # Lambda or Function node.
+            childnodes = childnodes[:len(ast.defaults)]
+
+        for child in childnodes:
+            if isinstance(ast, Node):
+                collect_captured(child, declared, captured)
+
+
+def make_result(funcnode, var_name, var_lineno):
+    if hasattr(funcnode, 'name'):
+        func_name = 'function %r' % (funcnode.name,)
+    else:
+        func_name = '<lambda>'
+    return (funcnode.lineno, func_name, var_name, var_lineno)
+
+def report(out, path, results):
+    for r in results:
+        if isinstance(r, SyntaxError):
+            print >>out, path + (" NOT ANALYSED due to syntax error: %s" % r)
+        else:
+            print >>out, path + (":%r %s captures %r declared at line %d" % r)
+
+def check(sources, out):
+    class Counts:
+        n = 0
+        processed_files = 0
+        suspect_files = 0
+    counts = Counts()
+
+    def _process(path):
+        results = check_file(path)
+        report(out, path, results)
+        counts.n += len(results)
+        counts.processed_files += 1
+        if len(results) > 0:
+            counts.suspect_files += 1
+
+    for source in sources:
+        print >>out, "Checking %s..." % (source,)
+        if os.path.isfile(source):
+            _process(source)
+        else:
+            for (dirpath, dirnames, filenames) in os.walk(source):
+                for fn in filenames:
+                    (basename, ext) = os.path.splitext(fn)
+                    if ext == '.py':
+                        _process(os.path.join(dirpath, fn))
+
+    print >>out, ("%d suspiciously captured variables in %d out of %d files"
+                  % (counts.n, counts.suspect_files, counts.processed_files))
+    return counts.n
+
+
+sources = ['src']
+if len(sys.argv) > 1:
+    sources = sys.argv[1:]
+if check(sources, sys.stderr) > 0:
+    sys.exit(1)
+
+
+# TODO: self-tests
}
[check-miscaptures.py: check while loops and list comprehensions as well as for loops. Also fix a pyflakes warning. refs #1555
david-sarah@jacaranda.org**20111009044022
 Ignore-this: 6526e4e315ca6461b1fbc2da5568e444
] {
hunk ./misc/coding_tools/check-miscaptures.py 3
 #! /usr/bin/python
 
-import os, sys, compiler, traceback
-from compiler.ast import Node, For, AssName, Name, Lambda, Function
+import os, sys, compiler
+from compiler.ast import Node, For, While, ListComp, AssName, Name, Lambda, Function
 
 
 def check_source(source):
hunk ./misc/coding_tools/check-miscaptures.py 24
         return results
 
 def check_ast(ast, results):
-    """Check a node outside a 'for' loop."""
-    if isinstance(ast, For):
-        check_for(ast, results)
+    """Check a node outside a loop."""
+    if isinstance(ast, (For, While, ListComp)):
+        check_loop(ast, results)
     else:
         for child in ast.getChildNodes():
             if isinstance(ast, Node):
hunk ./misc/coding_tools/check-miscaptures.py 32
                 check_ast(child, results)
 
-def check_for(ast, results):
-    """Check a particular outer 'for' loop."""
+def check_loop(ast, results):
+    """Check a particular outer loop."""
+
+    # List comprehensions have a poorly designed AST of the form
+    # ListComp(exprNode, [ListCompFor(...), ...]), in which the
+    # result expression is outside the ListCompFor node even though
+    # it is logically inside the loop(s).
+    # There may be multiple ListCompFor nodes (in cases such as
+    #   [lambda: (a,b) for a in ... for b in ...]
+    # ), and that case they are not nested in the AST. But these
+    # warts (nonobviously) happen not to matter for our analysis.
 
     declared = {}  # maps name to lineno of declaration
     nested = set()
}
[check-miscaptures.py: handle destructuring function arguments correctly. refs #1555
david-sarah@jacaranda.org**20111009044710
 Ignore-this: f9de7d95e94446507a206c88d3f98a23
] {
hunk ./misc/coding_tools/check-miscaptures.py 99
         if isinstance(ast, (Lambda, Function)):
             # Formal parameters of the function are excluded from
             # captures we care about in subnodes of the function body.
-            declared = declared.copy()
-            for argname in ast.argnames:
-                if argname in declared:
-                    del declared[argname]
+            new_declared = declared.copy()
+            remove_argnames(ast.argnames, new_declared)
 
             for child in childnodes[len(ast.defaults):]:
                 collect_captured(child, declared, captured)
hunk ./misc/coding_tools/check-miscaptures.py 115
                 collect_captured(child, declared, captured)
 
 
+def remove_argnames(names, fromset):
+    for element in names:
+        if element in fromset:
+            del fromset[element]
+        elif isinstance(element, (tuple, list)):
+            remove_argnames(element, fromset)
+
+
 def make_result(funcnode, var_name, var_lineno):
     if hasattr(funcnode, 'name'):
         func_name = 'function %r' % (funcnode.name,)
}
[check-miscaptures.py: Python doesn't really have declarations; report the topmost assignment. refs #1555
david-sarah@jacaranda.org**20111009044800
 Ignore-this: 4905c9dfe7726f433333e216a6760a4b
] {
hunk ./misc/coding_tools/check-miscaptures.py 44
     # ), and that case they are not nested in the AST. But these
     # warts (nonobviously) happen not to matter for our analysis.
 
-    declared = {}  # maps name to lineno of declaration
+    declared = {}  # maps name to lineno of topmost assignment
     nested = set()
     collect_declared_and_nested(ast, declared, nested)
 
hunk ./misc/coding_tools/check-miscaptures.py 56
         for name in captured:
             # We want to report the outermost capturing function
             # (since that is where the workaround will need to be
-            # added), and the variable declaration. Just one report
-            # per capturing function per variable will do.
+            # added), and the topmost assignment to the variable.
+            # Just one report per capturing function per variable
+            # will do.
             results.append(make_result(funcnode, name, declared[name]))
 
         # Check each node in the function body in case it
hunk ./misc/coding_tools/check-miscaptures.py 69
 
 def collect_declared_and_nested(ast, declared, nested):
     """
-    Collect the names declared in this 'for' loop, not including
-    names declared in nested functions. Also collect the nodes of
-    functions that are nested one level deep.
+    Collect the names declared in this loop, not including names
+    declared in nested functions. Also collect the nodes of functions
+    that are nested one level deep.
     """
     if isinstance(ast, AssName):
hunk ./misc/coding_tools/check-miscaptures.py 74
-        declared[ast.name] = ast.lineno
+        if ast.name not in declared or declared[ast.name] > ast.lineno:
+            declared[ast.name] = ast.lineno
     else:
         childnodes = ast.getChildNodes()
         if isinstance(ast, (Lambda, Function)):
hunk ./misc/coding_tools/check-miscaptures.py 97
             captured.add(ast.name)
     else:
         childnodes = ast.getChildNodes()
-
         if isinstance(ast, (Lambda, Function)):
             # Formal parameters of the function are excluded from
             # captures we care about in subnodes of the function body.
replace ./misc/coding_tools/check-miscaptures.py [A-Za-z_0-9] collect_declared_and_nested collect_assigned_and_nested
replace ./misc/coding_tools/check-miscaptures.py [A-Za-z_0-9] declared assigned
replace ./misc/coding_tools/check-miscaptures.py [A-Za-z_0-9] new_declared new_assigned
}
[check-miscaptures.py: handle corner cases around default arguments correctly. Also make a minor optimization when there are no assigned variables to consider. refs #1555
david-sarah@jacaranda.org**20111009045023
 Ignore-this: f49ece515620081da1d745ae6da19d21
] {
hunk ./misc/coding_tools/check-miscaptures.py 52
     for funcnode in nested:
         # Check for captured variables in this function.
         captured = set()
-        collect_captured(funcnode, assigned, captured)
+        collect_captured(funcnode, assigned, captured, False)
         for name in captured:
             # We want to report the outermost capturing function
             # (since that is where the workaround will need to be
hunk ./misc/coding_tools/check-miscaptures.py 90
             if isinstance(ast, Node):
                 collect_assigned_and_nested(child, assigned, nested)
 
-def collect_captured(ast, assigned, captured):
+def collect_captured(ast, assigned, captured, in_function_yet):
     """Collect any captured variables that are also in assigned."""
     if isinstance(ast, Name):
         if ast.name in assigned:
hunk ./misc/coding_tools/check-miscaptures.py 103
             new_assigned = assigned.copy()
             remove_argnames(ast.argnames, new_assigned)
 
-            for child in childnodes[len(ast.defaults):]:
-                collect_captured(child, assigned, captured)
+            if len(new_assigned) > 0:
+                for child in childnodes[len(ast.defaults):]:
+                    collect_captured(child, new_assigned, captured, True)
 
hunk ./misc/coding_tools/check-miscaptures.py 107
-            # The default argument expressions are "outside" the
-            # function, even though they are children of the
-            # Lambda or Function node.
+            # The default argument expressions are "outside" *this*
+            # function, even though they are children of the Lambda or
+            # Function node.
+            if not in_function_yet:
+                return
             childnodes = childnodes[:len(ast.defaults)]
 
         for child in childnodes:
hunk ./misc/coding_tools/check-miscaptures.py 116
             if isinstance(ast, Node):
-                collect_captured(child, assigned, captured)
+                collect_captured(child, assigned, captured, True)
 
 
 def remove_argnames(names, fromset):
}
[check-miscaptures.py: report the number of files that were not analysed due to syntax errors (and don't count them in the number of suspicious captures). refs #1555
david-sarah@jacaranda.org**20111009050301
 Ignore-this: 62ee03f4b8a96c292e75c097ad87d52e
] {
hunk ./misc/coding_tools/check-miscaptures.py 17
     try:
         ast = parser(thing)
     except SyntaxError, e:
-        return [e]
+        return e
     else:
         results = []
         check_ast(ast, results)
hunk ./misc/coding_tools/check-miscaptures.py 136
 
 def report(out, path, results):
     for r in results:
-        if isinstance(r, SyntaxError):
-            print >>out, path + (" NOT ANALYSED due to syntax error: %s" % r)
-        else:
-            print >>out, path + (":%r %s captures %r assigned at line %d" % r)
+        print >>out, path + (":%r %s captures %r assigned at line %d" % r)
 
 def check(sources, out):
     class Counts:
hunk ./misc/coding_tools/check-miscaptures.py 143
         n = 0
         processed_files = 0
         suspect_files = 0
+        error_files = 0
     counts = Counts()
 
     def _process(path):
hunk ./misc/coding_tools/check-miscaptures.py 148
         results = check_file(path)
-        report(out, path, results)
-        counts.n += len(results)
-        counts.processed_files += 1
-        if len(results) > 0:
-            counts.suspect_files += 1
+        if isinstance(results, SyntaxError):
+            print >>out, path + (" NOT ANALYSED due to syntax error: %s" % results)
+            counts.error_files += 1
+        else:
+            report(out, path, results)
+            counts.n += len(results)
+            counts.processed_files += 1
+            if len(results) > 0:
+                counts.suspect_files += 1
 
     for source in sources:
         print >>out, "Checking %s..." % (source,)
hunk ./misc/coding_tools/check-miscaptures.py 169
                     if ext == '.py':
                         _process(os.path.join(dirpath, fn))
 
-    print >>out, ("%d suspiciously captured variables in %d out of %d files"
+    print >>out, ("%d suspiciously captured variables in %d out of %d file(s)."
                   % (counts.n, counts.suspect_files, counts.processed_files))
hunk ./misc/coding_tools/check-miscaptures.py 171
+    if counts.error_files > 0:
+        print >>out, ("%d file(s) not processed due to syntax errors."
+                      % (counts.error_files,))
     return counts.n
 
 
}

Context:

[immutable/literal.py: add pauseProducing method to LiteralProducer. refs #1537
david-sarah@jacaranda.org**20111003195239
 Ignore-this: 385ee3379a2819381937357f1eac457
] 
[no_network.py: Clean up whitespace around code changed by previous patch.
david-sarah@jacaranda.org**20111004010407
 Ignore-this: 647ec8a9346dca1a41212ab250619b72
] 
[no_network.py: Fix potential bugs in some tests due to capture of slots in for loops.
david-sarah@jacaranda.org**20111004010231
 Ignore-this: 9c496877613a3befd54979e5de6e63d2
] 
[test/test_runner.py: BinTahoe.test_path has rare nondeterministic failures; this patch probably fixes a problem where the actual cause of failure is masked by a string conversion error.
david-sarah@jacaranda.org**20110927225336
 Ignore-this: 6f1ad68004194cc9cea55ace3745e4af
] 
[docs/configuration.rst: add section about the types of node, and clarify when setting web.port enables web-API service. fixes #1444
zooko@zooko.com**20110926203801
 Ignore-this: ab94d470c68e720101a7ff3c207a719e
] 
[TAG allmydata-tahoe-1.9.0a2
warner@lothar.com**20110925234811
 Ignore-this: e9649c58f9c9017a7d55008938dba64f
] 
[NEWS: tidy up a little bit, reprioritize some items, hide some non-user-visible items
warner@lothar.com**20110925233529
 Ignore-this: 61f334cc3fa2539742c3e5d2801aee81
] 
[docs: fix some broken .rst links. refs #1542
david-sarah@jacaranda.org**20110925051001
 Ignore-this: 5714ee650abfcaab0914537e1f206972
] 
[mutable/publish.py: fix an unused import. refs #1542
david-sarah@jacaranda.org**20110925052206
 Ignore-this: 2d69ac9e605e789c0aedfecb8877b7d7
] 
[NEWS: fix .rst formatting.
david-sarah@jacaranda.org**20110925050119
 Ignore-this: aa1d20acd23bdb8f8f6d0fa048ea0277
] 
[NEWS: updates for 1.9alpha2.
david-sarah@jacaranda.org**20110925045343
 Ignore-this: d2c44e4e05d2ed662b7adfd2e43928bc
] 
[mutable/layout.py: make unpack_sdmf_checkstring and unpack_mdmf_checkstring more similar, and change an assert to give a more useful message if it fails. refs #1540
david-sarah@jacaranda.org**20110925023651
 Ignore-this: 977aaa8cb16e06a6dcc3e27cb6e23956
] 
[mutable/publish: handle unknown mutable share formats when handling errors
kevan@isnotajoke.com**20110925004305
 Ignore-this: 4d5fa44ef7d777c432eb10c9584ad51f
] 
[mutable/layout: break unpack_checkstring into unpack_mdmf_checkstring and unpack_sdmf_checkstring, add distinguisher function for checkstrings
kevan@isnotajoke.com**20110925004134
 Ignore-this: 57f49ed5a72e418a69c7286a225cc8fb
] 
[test/test_mutable: reenable mdmf publish surprise test
kevan@isnotajoke.com**20110924235415
 Ignore-this: f752e47a703684491305cc83d16248fb
] 
[mutable/publish: use unpack_mdmf_checkstring and unpack_sdmf_checkstring instead of unpack_checkstring. fixes #1540
kevan@isnotajoke.com**20110924235137
 Ignore-this: 52ca3d9627b8b0ba758367b2bd6c7085
] 
[misc/coding_tools/check_interfaces.py: report all violations rather than only one for a given class, by including a forked version of verifyClass. refs #1474
david-sarah@jacaranda.org**20110916223450
 Ignore-this: 927efeecf4d12588316826a4b3479aa9
] 
[misc/coding_tools/check_interfaces.py: use os.walk instead of FilePath, since this script shouldn't really depend on Twisted. refs #1474
david-sarah@jacaranda.org**20110916212633
 Ignore-this: 46eeb4236b34375227dac71ef53f5428
] 
[misc/coding_tools/check-interfaces.py: reduce false-positives by adding Dummy* to the set of excluded classnames, and bench-* to the set of excluded basenames. refs #1474
david-sarah@jacaranda.org**20110916212624
 Ignore-this: 4e78f6e6fe6c0e9be9df826a0e206804
] 
[Add a script 'misc/coding_tools/check-interfaces.py' that checks whether zope interfaces are enforced. Also add 'check-interfaces', 'version-and-path', and 'code-checks' targets to the Makefile. fixes #1474
david-sarah@jacaranda.org**20110915161532
 Ignore-this: 32d9bdc5bc4a86d21e927724560ad4b4
] 
[mutable/publish.py: copy the self.writers dict before iterating over it, since we remove elements from it during the iteration. refs #393
david-sarah@jacaranda.org**20110924211208
 Ignore-this: 76d4066b55d50ace2a34b87443b39094
] 
[mutable/publish.py: simplify by refactoring self.outstanding to self.num_outstanding. refs #393
david-sarah@jacaranda.org**20110924205004
 Ignore-this: 902768cfc529ae13ae0b7f67768a3643
] 
[test_mutable.py: update SkipTest message for test_publish_surprise_mdmf to reference the right ticket number. refs #1540.
david-sarah@jacaranda.org**20110923211622
 Ignore-this: 44f16a6817a6b75930bbba18b0a516be
] 
[control.py: unbreak speed-test: overwrite() wants a MutableData, not str
Brian Warner <warner@lothar.com>**20110923073748
 Ignore-this: 7dad7aff3d66165868a64ae22d225fa3
 
 Really, all the upload/modify APIs should take a string or a filehandle, and
 internally wrap it as needed. Callers should not need to be aware of
 Uploadable() or MutableData() classes.
] 
[test_mutable.py: skip test_publish_surprise_mdmf, which is causing an error. refs #1534, #393
david-sarah@jacaranda.org**20110920183319
 Ignore-this: 6fb020e09e8de437cbcc2c9f57835b31
] 
[test/test_mutable: write publish surprise test for MDMF, rename existing test_publish_surprise to clarify that it is for SDMF
kevan@isnotajoke.com**20110918003657
 Ignore-this: 722c507e8f5b537ff920e0555951059a
] 
[test/test_mutable: refactor publish surprise test into common test fixture, rewrite test_publish_surprise to use test fixture
kevan@isnotajoke.com**20110918003533
 Ignore-this: 6f135888d400a99a09b5f9a4be443b6e
] 
[mutable/publish: add errback immediately after write, don't consume errors from other parts of the publisher
kevan@isnotajoke.com**20110917234708
 Ignore-this: 12bf6b0918a5dc5ffc30ece669fad51d
] 
[.darcs-boringfile: minor cleanups.
david-sarah@jacaranda.org**20110920154918
 Ignore-this: cab78e30d293da7e2832207dbee2ffeb
] 
[uri.py: fix two interface violations in verifier URI classes. refs #1474
david-sarah@jacaranda.org**20110920030156
 Ignore-this: 454ddd1419556cb1d7576d914cb19598
] 
[Make platform-detection code tolerate linux-3.0, patch by zooko.
Brian Warner <warner@lothar.com>**20110915202620
 Ignore-this: af63cf9177ae531984dea7a1cad03762
 
 Otherwise address-autodetection can't find ifconfig. refs #1536
] 
[test_web.py: fix a bug in _count_leases that was causing us to check only the lease count of one share file, not of all share files as intended.
david-sarah@jacaranda.org**20110915185126
 Ignore-this: d96632bc48d770b9b577cda1bbd8ff94
] 
[docs: insert a newline at the beginning of known_issues.rst to see if this makes it render more nicely in trac
zooko@zooko.com**20110914064728
 Ignore-this: aca15190fa22083c5d4114d3965f5d65
] 
[docs: remove the coding: utf-8 declaration at the to of known_issues.rst, since the trac rendering doesn't hide it
zooko@zooko.com**20110914055713
 Ignore-this: 941ed32f83ead377171aa7a6bd198fcf
] 
[docs: more cleanup of known_issues.rst -- now it passes "rst2html --verbose" without comment
zooko@zooko.com**20110914055419
 Ignore-this: 5505b3d76934bd97d0312cc59ed53879
] 
[docs: more formatting improvements to known_issues.rst
zooko@zooko.com**20110914051639
 Ignore-this: 9ae9230ec9a38a312cbacaf370826691
] 
[docs: reformatting of known_issues.rst
zooko@zooko.com**20110914050240
 Ignore-this: b8be0375079fb478be9d07500f9aaa87
] 
[docs: fix formatting error in docs/known_issues.rst
zooko@zooko.com**20110914045909
 Ignore-this: f73fe74ad2b9e655aa0c6075acced15a
] 
[merge Tahoe-LAFS v1.8.3 release announcement with trunk
zooko@zooko.com**20110913210544
 Ignore-this: 163f2c3ddacca387d7308e4b9332516e
] 
[docs: release notes for Tahoe-LAFS v1.8.3
zooko@zooko.com**20110913165826
 Ignore-this: 84223604985b14733a956d2fbaeb4e9f
] 
[tests: bump up the timeout in this test that fails on FreeStorm's CentOS in order to see if it is just very slow
zooko@zooko.com**20110913024255
 Ignore-this: 6a86d691e878cec583722faad06fb8e4
] 
[interfaces: document that the 'fills-holes-with-zero-bytes' key should be used to detect whether a storage server has that behavior. refs #1528
david-sarah@jacaranda.org**20110913002843
 Ignore-this: 1a00a6029d40f6792af48c5578c1fd69
] 
[CREDITS: more CREDITS for Kevan and David-Sarah
zooko@zooko.com**20110912223357
 Ignore-this: 4ea8f0d6f2918171d2f5359c25ad1ada
] 
[merge NEWS about the mutable file bounds fixes with NEWS about work-in-progress
zooko@zooko.com**20110913205521
 Ignore-this: 4289a4225f848d6ae6860dd39bc92fa8
] 
[doc: add NEWS item about fixes to potential palimpsest issues in mutable files
zooko@zooko.com**20110912223329
 Ignore-this: 9d63c95ddf95c7d5453c94a1ba4d406a
 ref. #1528
] 
[merge the NEWS about the security fix (#1528) with the work-in-progress NEWS
zooko@zooko.com**20110913205153
 Ignore-this: 88e88a2ad140238c62010cf7c66953fc
] 
[doc: add NEWS entry about the issue which allows unauthorized deletion of shares
zooko@zooko.com**20110912223246
 Ignore-this: 77e06d09103d2ef6bb51ea3e5d6e80b0
 ref. #1528
] 
[doc: add entry in known_issues.rst about the issue which allows unauthorized deletion of shares
zooko@zooko.com**20110912223135
 Ignore-this: b26c6ea96b6c8740b93da1f602b5a4cd
 ref. #1528
] 
[storage: more paranoid handling of bounds and palimpsests in mutable share files
zooko@zooko.com**20110912222655
 Ignore-this: a20782fa423779ee851ea086901e1507
 * storage server ignores requests to extend shares by sending a new_length
 * storage server fills exposed holes (created by sending a write vector whose offset begins after the end of the current data) with 0 to avoid "palimpsest" exposure of previous contents
 * storage server zeroes out lease info at the old location when moving it to a new location
 ref. #1528
] 
[storage: test that the storage server ignores requests to extend shares by sending a new_length, and that the storage server fills exposed holes with 0 to avoid "palimpsest" exposure of previous contents
zooko@zooko.com**20110912222554
 Ignore-this: 61ebd7b11250963efdf5b1734a35271
 ref. #1528
] 
[immutable: prevent clients from reading past the end of share data, which would allow them to learn the cancellation secret
zooko@zooko.com**20110912222458
 Ignore-this: da1ebd31433ea052087b75b2e3480c25
 Declare explicitly that we prevent this problem in the server's version dict.
 fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
] 
[storage: remove the storage server's "remote_cancel_lease" function
zooko@zooko.com**20110912222331
 Ignore-this: 1c32dee50e0981408576daffad648c50
 We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
 fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
] 
[storage: test that the storage server does *not* have a "remote_cancel_lease" function
zooko@zooko.com**20110912222324
 Ignore-this: 21c652009704652d35f34651f98dd403
 We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
 ref. #1528
] 
[immutable: test whether the server allows clients to read past the end of share data, which would allow them to learn the cancellation secret
zooko@zooko.com**20110912221201
 Ignore-this: 376e47b346c713d37096531491176349
 Also test whether the server explicitly declares that it prevents this problem.
 ref #1528
] 
[Retrieve._activate_enough_peers: rewrite Verify logic
Brian Warner <warner@lothar.com>**20110909181150
 Ignore-this: 9367c11e1eacbf025f75ce034030d717
] 
[Retrieve: implement/test stopProducing
Brian Warner <warner@lothar.com>**20110909181150
 Ignore-this: 47b2c3df7dc69835e0a066ca12e3c178
] 
[move DownloadStopped from download.common to interfaces
Brian Warner <warner@lothar.com>**20110909181150
 Ignore-this: 8572acd3bb16e50341dbed8eb1d90a50
] 
[retrieve.py: remove vestigal self._validated_readers
Brian Warner <warner@lothar.com>**20110909181150
 Ignore-this: faab2ec14e314a53a2ffb714de626e2d
] 
[Retrieve: rewrite flow-control: use a top-level loop() to catch all errors
Brian Warner <warner@lothar.com>**20110909181150
 Ignore-this: e162d2cd53b3d3144fc6bc757e2c7714
 
 This ought to close the potential for dropped errors and hanging downloads.
 Verify needs to be examined, I may have broken it, although all tests pass.
] 
[Retrieve: merge _validate_active_prefixes into _add_active_peers
Brian Warner <warner@lothar.com>**20110909181150
 Ignore-this: d3ead31e17e69394ae7058eeb5beaf4c
] 
[Retrieve: remove the initial prefix-is-still-good check
Brian Warner <warner@lothar.com>**20110909181150
 Ignore-this: da66ee51c894eaa4e862e2dffb458acc
 
 This check needs to be done with each fetch from the storage server, to
 detect when someone has changed the share (i.e. our servermap goes stale).
 Doing it just once at the beginning of retrieve isn't enough: a write might
 occur after the first segment but before the second, etc.
 
 _try_to_validate_prefix() was not removed: it will be used by the future
 check-with-each-fetch code.
 
 test_mutable.Roundtrip.test_corrupt_all_seqnum_late was disabled, since it
 fails until this check is brought back. (the corruption it applies only
 touches the prefix, not the block data, so the check-less retrieve actually
 tolerates it). Don't forget to re-enable it once the check is brought back.
] 
[MDMFSlotReadProxy: remove the queue
Brian Warner <warner@lothar.com>**20110909181150
 Ignore-this: 96673cb8dda7a87a423de2f4897d66d2
 
 This is a neat trick to reduce Foolscap overhead, but the need for an
 explicit flush() complicates the Retrieve path and makes it prone to
 lost-progress bugs.
 
 Also change test_mutable.FakeStorageServer to tolerate multiple reads of the
 same share in a row, a limitation exposed by turning off the queue.
] 
[rearrange Retrieve: first step, shouldn't change order of execution
Brian Warner <warner@lothar.com>**20110909181149
 Ignore-this: e3006368bfd2802b82ea45c52409e8d6
] 
[CLI: test_cli.py -- remove an unnecessary call in test_mkdir_mutable_type. refs #1527
david-sarah@jacaranda.org**20110906183730
 Ignore-this: 122e2ffbee84861c32eda766a57759cf
] 
[CLI: improve test for 'tahoe mkdir --mutable-type='. refs #1527
david-sarah@jacaranda.org**20110906183020
 Ignore-this: f1d4598e6c536f0a2b15050b3bc0ef9d
] 
[CLI: make the --mutable-type option value for 'tahoe put' and 'tahoe mkdir' case-insensitive, and change --help for these commands accordingly. fixes #1527
david-sarah@jacaranda.org**20110905020922
 Ignore-this: 75a6df0a2df9c467d8c010579e9a024e
] 
[cli: make --mutable-type imply --mutable in 'tahoe put'
Kevan Carstensen <kevan@isnotajoke.com>**20110903190920
 Ignore-this: 23336d3c43b2a9554e40c2a11c675e93
] 
[SFTP: add a comment about a subtle interaction between OverwriteableFileConsumer and GeneralSFTPFile, and test the case it is commenting on.
david-sarah@jacaranda.org**20110903222304
 Ignore-this: 980c61d4dd0119337f1463a69aeebaf0
] 
[improve the storage/mutable.py asserts even more
warner@lothar.com**20110901160543
 Ignore-this: 5b2b13c49bc4034f96e6e3aaaa9a9946
] 
[storage/mutable.py: special characters in struct.foo arguments indicate standard as opposed to native sizes, we should be using these characters in these asserts
wilcoxjg@gmail.com**20110901084144
 Ignore-this: 28ace2b2678642e4d7269ddab8c67f30
] 
[docs/write_coordination.rst: fix formatting and add more specific warning about access via sshfs.
david-sarah@jacaranda.org**20110831232148
 Ignore-this: cd9c851d3eb4e0a1e088f337c291586c
] 
[test_mutable.Version: consolidate some tests, reduce runtime from 19s to 15s
warner@lothar.com**20110831050451
 Ignore-this: 64815284d9e536f8f3798b5f44cf580c
] 
[mutable/retrieve: handle the case where self._read_length is 0.
Kevan Carstensen <kevan@isnotajoke.com>**20110830210141
 Ignore-this: fceafbe485851ca53f2774e5a4fd8d30
 
 Note that the downloader will still fetch a segment for a zero-length
 read, which is wasteful. Fixing that isn't specifically required to fix
 #1512, but it should probably be fixed before 1.9.
] 
[NEWS: added summary of all changes since 1.8.2. Needs editing.
Brian Warner <warner@lothar.com>**20110830163205
 Ignore-this: 273899b37a899fc6919b74572454b8b2
] 
[test_mutable.Update: only upload the files needed for each test. refs #1500
Brian Warner <warner@lothar.com>**20110829072717
 Ignore-this: 4d2ab4c7523af9054af7ecca9c3d9dc7
 
 This first step shaves 15% off the runtime: from 139s to 119s on my laptop.
 It also fixes a couple of places where a Deferred was being dropped, which
 would cause two tests to run in parallel and also confuse error reporting.
] 
[Let Uploader retain History instead of passing it into upload(). Fixes #1079.
Brian Warner <warner@lothar.com>**20110829063246
 Ignore-this: 3902c58ec12bd4b2d876806248e19f17
 
 This consistently records all immutable uploads in the Recent Uploads And
 Downloads page, regardless of code path. Previously, certain webapi upload
 operations (like PUT /uri/$DIRCAP/newchildname) failed to pass the History
 object and were left out.
] 
[Fix mutable publish/retrieve timing status displays. Fixes #1505.
Brian Warner <warner@lothar.com>**20110828232221
 Ignore-this: 4080ce065cf481b2180fd711c9772dd6
 
 publish:
 * encrypt and encode times are cumulative, not just current-segment
 
 retrieve:
 * same for decrypt and decode times
 * update "current status" to include segment number
 * set status to Finished/Failed when download is complete
 * set progress to 1.0 when complete
 
 More improvements to consider:
 * progress is currently 0% or 100%: should calculate how many segments are
   involved (remembering retrieve can be less than the whole file) and set it
   to a fraction
 * "fetch" time is fuzzy: what we want is to know how much of the delay is not
   our own fault, but since we do decode/decrypt work while waiting for more
   shares, it's not straightforward
] 
[Teach 'tahoe debug catalog-shares about MDMF. Closes #1507.
Brian Warner <warner@lothar.com>**20110828080931
 Ignore-this: 56ef2951db1a648353d7daac6a04c7d1
] 
[debug.py: remove some dead comments
Brian Warner <warner@lothar.com>**20110828074556
 Ignore-this: 40e74040dd4d14fd2f4e4baaae506b31
] 
[hush pyflakes
Brian Warner <warner@lothar.com>**20110828074254
 Ignore-this: bef9d537a969fa82fe4decc4ba2acb09
] 
[MutableFileNode.set_downloader_hints: never depend upon order of dict.values()
Brian Warner <warner@lothar.com>**20110828074103
 Ignore-this: caaf1aa518dbdde4d797b7f335230faa
 
 The old code was calculating the "extension parameters" (a list) from the
 downloader hints (a dictionary) with hints.values(), which is not stable, and
 would result in corrupted filecaps (with the 'k' and 'segsize' hints
 occasionally swapped). The new code always uses [k,segsize].
] 
[layout.py: fix MDMF share layout documentation
Brian Warner <warner@lothar.com>**20110828073921
 Ignore-this: 3f13366fed75b5e31b51ae895450a225
] 
[teach 'tahoe debug dump-share' about MDMF and offsets. refs #1507
Brian Warner <warner@lothar.com>**20110828073834
 Ignore-this: 3a9d2ef9c47a72bf1506ba41199a1dea
] 
[test_mutable.Version.test_debug: use splitlines() to fix buildslaves
Brian Warner <warner@lothar.com>**20110828064728
 Ignore-this: c7f6245426fc80b9d1ae901d5218246a
 
 Any slave running in a directory with spaces in the name was miscounting
 shares, causing the test to fail.
] 
[test_mutable.Version: exercise 'tahoe debug find-shares' on MDMF. refs #1507
Brian Warner <warner@lothar.com>**20110828005542
 Ignore-this: cb20bea1c28bfa50a72317d70e109672
 
 Also changes NoNetworkGrid to put shares in storage/shares/ .
] 
[test_mutable.py: oops, missed a .todo
Brian Warner <warner@lothar.com>**20110828002118
 Ignore-this: fda09ae86481352b7a627c278d2a3940
] 
[test_mutable: merge davidsarah's patch with my Version refactorings
warner@lothar.com**20110827235707
 Ignore-this: b5aaf481c90d99e33827273b5d118fd0
] 
[Make the immutable/read-only constraint checking for MDMF URIs identical to that for SSK URIs. refs #393
david-sarah@jacaranda.org**20110823012720
 Ignore-this: e1f59d7ff2007c81dbef2aeb14abd721
] 
[Additional tests for MDMF URIs and for zero-length files. refs #393
david-sarah@jacaranda.org**20110823011532
 Ignore-this: a7cc0c09d1d2d72413f9cd227c47a9d5
] 
[Additional tests for zero-length partial reads and updates to mutable versions. refs #393
david-sarah@jacaranda.org**20110822014111
 Ignore-this: 5fc6f4d06e11910124e4a277ec8a43ea
] 
[test_mutable.Version: factor out some expensive uploads, save 25% runtime
Brian Warner <warner@lothar.com>**20110827232737
 Ignore-this: ea37383eb85ea0894b254fe4dfb45544
] 
[SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.
Brian Warner <warner@lothar.com>**20110827225031
 Ignore-this: b50ae6e1045818c400079f118b4ef48
 
 Without this, we get a regression when modifying a mutable file that was
 created with more shares (larger N) than our current tahoe.cfg . The
 modification attempt creates new versions of the (0,1,..,newN-1) shares, but
 leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
 assertion error in SDMFSlotWriteProxy.finish_publishing in the process).
 
 The mixed versions that result (some shares with e.g. N=10, some with N=20,
 such that both versions are recoverable) cause problems for the Publish code,
 even before MDMF landed. Might be related to refs #1390 and refs #1042.
] 
[layout.py: annotate assertion to figure out 'tahoe backup' failure
Brian Warner <warner@lothar.com>**20110827195253
 Ignore-this: 9b92b954e3ed0d0f80154fff1ff674e5
] 
[Add 'tahoe debug dump-cap' support for MDMF, DIR2-CHK, DIR2-MDMF. refs #1507.
Brian Warner <warner@lothar.com>**20110827195048
 Ignore-this: 61c6af5e33fc88e0251e697a50addb2c
 
 This also adds tests for all those cases, and fixes an omission in uri.py
 that broke parsing of DIR2-MDMF-Verifier and DIR2-CHK-Verifier.
] 
[MDMF: more writable/writeable consistentifications
warner@lothar.com**20110827190602
 Ignore-this: 22492a9e20c1819ddb12091062888b55
] 
[MDMF: s/Writable/Writeable/g, for consistency with existing SDMF code
warner@lothar.com**20110827183357
 Ignore-this: 9dd312acedbdb2fc2f7bef0d0fb17c0b
] 
[setup.cfg: remove no-longer-supported test_mac_diskimage alias. refs #1479
david-sarah@jacaranda.org**20110826230345
 Ignore-this: 40e908b8937322a290fb8012bfcad02a
] 
[test_mutable.Update: increase timeout from 120s to 400s, slaves are failing
Brian Warner <warner@lothar.com>**20110825230140
 Ignore-this: 101b1924a30cdbda9b2e419e95ca15ec
] 
[tests: fix check_memory test
zooko@zooko.com**20110825201116
 Ignore-this: 4d66299fa8cb61d2ca04b3f45344d835
 fixes #1503
] 
[TAG allmydata-tahoe-1.9.0a1
warner@lothar.com**20110825161122
 Ignore-this: 3cbf49f00dbda58189f893c427f65605
] 
Patch bundle hash:
b48f36aa306bef5a2df8b972db237a44a6f56aab