Ticket #1555: check-miscaptures.darcs.patch

1 patch for repository http://tahoe-lafs.org/source/tahoe/trunk:

Fri Oct  7 03:15:29 BST 2011  david-sarah@jacaranda.org
  * Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) refs #1555

New patches:

[Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) refs #1555
david-sarah@jacaranda.org**20111007021529
 Ignore-this: 212f4f6039af508716e09b8952ae7ab
] {
hunk ./Makefile 124
        false
 endif
 
-code-checks: build version-and-path check-interfaces -find-trailing-spaces -check-umids pyflakes
+code-checks: build version-and-path check-interfaces check-miscaptures -find-trailing-spaces -check-umids pyflakes
 
 version-and-path:
        $(TAHOE) --version-and-path
hunk ./Makefile 133
        $(TAHOE) @misc/coding_tools/check-interfaces.py 2>&1 |tee violations.txt
        @echo
 
+check-miscaptures:
+       $(PYTHON) misc/coding_tools/check-miscaptures.py $(SOURCES) 2>&1 |tee miscaptures.txt
+       @echo
+
 pyflakes:
        $(PYTHON) -OOu `which pyflakes` $(SOURCES) |sort |uniq
        @echo
addfile ./misc/coding_tools/check-miscaptures.py
hunk ./misc/coding_tools/check-miscaptures.py 1
+#! /usr/bin/python
+
+import os, sys, compiler
+from compiler.ast import Node, For, AssName, Name, Lambda, Function
+
+def check_file(filename):
+    results = []
+    check_ast(compiler.parseFile(filename), results)
+    return results
+
+def check_source(source):
+    results = []
+    check_ast(compiler.parse(source), results)
+    return results
+
+def check_ast(ast, results):
+    if isinstance(ast, For):
+        check_for(ast, results)
+    else:
+        for child in ast.getChildNodes():
+            if isinstance(ast, Node):
+                check_ast(child, results)
+
+def check_for(ast, results):
+    declared = set()
+    captured = set()
+    collect_declared(ast, declared)
+    collect_captured(ast, captured, set(), False)
+    for name in captured & declared:
+        results.append((name, ast.lineno))
+
+def collect_declared(ast, declared):
+    if isinstance(ast, AssName):
+        declared.add(ast.name)
+    else:
+        for child in ast.getChildNodes():
+            if isinstance(ast, Node):
+                collect_declared(child, declared)
+
+def collect_captured(ast, captured, excluded, in_func):
+    if isinstance(ast, Name):
+        if in_func and ast.name not in excluded:
+            captured.add(ast.name)
+    elif isinstance(ast, (Lambda, Function)):
+        # Formal parameters of the function are not captures
+        newly_excluded = excluded.copy()
+        for argname in ast.argnames:
+            newly_excluded.add(argname)
+
+        childnodes = ast.getChildNodes()
+        # ... and neither are variable uses in default argument
+        # expressions, provided we weren't already in a function.
+        if not in_func:
+            childnodes = childnodes[len(ast.defaults):]
+
+        for child in childnodes:
+            if isinstance(ast, Node):
+                collect_captured(child, captured, newly_excluded, True)
+    else:
+        for child in ast.getChildNodes():
+            if isinstance(ast, Node):
+                collect_captured(child, captured, excluded, in_func)
+
+def report(path, results, out):
+    for (name, lineno) in results:
+        print >>out, "%s:%d %r" % (path, lineno, name)
+
+
+def check(sources, out):
+    class Counts:
+        n = 0
+        processed_files = 0
+        suspect_files = 0
+    counts = Counts()
+
+    def _process(path):
+        results = check_file(path)
+        report(path, results, out)
+        counts.n += len(results)
+        counts.processed_files += 1
+        if len(results) > 0:
+            counts.suspect_files += 1
+
+    for source in sources:
+        print >>out, "Checking %s..." % (source,)
+        if os.path.isfile(source):
+            _process(source)
+        else:
+            for (dirpath, dirnames, filenames) in os.walk(source):
+                for fn in filenames:
+                    (basename, ext) = os.path.splitext(fn)
+                    if ext == '.py':
+                        _process(os.path.join(dirpath, fn))
+
+    print >>out, ("%d suspiciously captured variables in %d out of %d files"
+                  % (counts.n, counts.suspect_files, counts.processed_files))
+
+
+sources = ['src']
+if len(sys.argv) > 1:
+    sources = sys.argv[1:]
+check(sources, sys.stderr)
+
+
+# TODO: self-tests
}

Context:

[no_network.py: Clean up whitespace around code changed by previous patch.
david-sarah@jacaranda.org**20111004010407
 Ignore-this: 647ec8a9346dca1a41212ab250619b72
] 
[no_network.py: Fix potential bugs in some tests due to capture of slots in for loops.
david-sarah@jacaranda.org**20111004010231
 Ignore-this: 9c496877613a3befd54979e5de6e63d2
] 
[docs: fix the rst formatting of COPYING.TGPPL.rst
zooko@zooko.com**20111003043333
 Ignore-this: c5fbc83f4a3db81a0c95b27053c463c5
 Now it renders correctly both on trac and with rst2html --verbose from docutils v0.8.1.
] 
[MDMF: remove extension fields from caps, tolerate arbitrary ones. Fixes #1526
Brian Warner <warner@lothar.com>**20111001233553
 Ignore-this: 335e1690aef1146a2c0b8d8c18c1cb21
 
 The filecaps used to be produced with hints for 'k' and segsize, but they
 weren't actually used, and doing so had the potential to limit how we change
 those filecaps in the future. Also the parsing code had some problems dealing
 with other numbers of extensions. Removing the existing fields and making the
 parser tolerate (and ignore) extra ones makes MDMF more future-proof.
] 
[test/test_runner.py: BinTahoe.test_path has rare nondeterministic failures; this patch probably fixes a problem where the actual cause of failure is masked by a string conversion error.
david-sarah@jacaranda.org**20110927225336
 Ignore-this: 6f1ad68004194cc9cea55ace3745e4af
] 
[docs/configuration.rst: add section about the types of node, and clarify when setting web.port enables web-API service. fixes #1444
zooko@zooko.com**20110926203801
 Ignore-this: ab94d470c68e720101a7ff3c207a719e
] 
[TAG allmydata-tahoe-1.9.0a2
warner@lothar.com**20110925234811
 Ignore-this: e9649c58f9c9017a7d55008938dba64f
] 
Patch bundle hash:
83fda893e4e4585dd65a793d1d45225f7b8177ee