add check-miscaptures.py script to check for incorrect variable captures in loops

[davidsarah]

The attached script checks for errors in which a variable declaredmodified in a for loop is captured by a lambda or function definition. If the lambda/function is executed after the loop iteration in which it was created, it will refer to some later value of the variable, which is usually not what was intended.

The script currently produces false positives if a variable is declared and used in the same lambda/function within a for loop (i.e. it is not captured). I don't intend to fix that until after the Tahoe summit in November, probably. However, it produces relatively few false positives already, and found some genuine bugs (see #1556).

[davidsarah]

Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) refs #1555

[davidsarah]

Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) This is a rewritten version that reports much fewer false positives, by determining captured variables more accurately. fixes #1555

[davidsarah]

Gahh, there's a pyflakes warning (for import traceback). Ignore that, I'll fix it before committing.

[david-sarah@…]

In [5401/ticket999-S3-backend]:

Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) This is a rewritten version that reports much fewer false positives, by determining captured variables more accurately. fixes #1555

[davidsarah]

(reopened after auto-close on a branch)

[davidsarah]

Various improvements: also check while loops and list comprehensions; fewer false positives; say "assigned" instance of "declared"; report number of files not analysed.

[davidsarah]

The remaining significant false positive cases are where:

• the capturing function is only used in the same iteration as the values it was intended to capture. For example, in

  for x in xs:
      print map(lambda y: x+y, ys)
  

  the use of each lambda is only in the same iteration as the x it was intended to capture (and map does not retain the function), so there is no bug.
• the captured variables are only assigned once, before any application of a capturing function. A somewhat artificial example:

  fs = []
  z = None
  for x in range(2):
      if z is None:
          z = 42
      fs.append(lambda y: y+z)
  

  because the captured z is effectively constant, even though it is assigned in the loop.

Both these cases are undecidable to detect in general. You could detect special cases of the first, such as builtin map or filter with a lambda expression as its first argument, but it's probably not worth the complication.

After applying the patches on #1556, Tahoe produces no warnings from make check-miscaptures, so I'd like to include that in the buildbot flow in the same way as pyflakes. (It would be ok to run it in the same step as pyflakes.)

[david-sarah@…]

In [5439/ticket999-S3-backend]:

check-miscaptures.py: check while loops and list comprehensions as well as for loops. Also fix a pyflakes warning. refs #1555

[david-sarah@…]

In [5440/ticket999-S3-backend]:

check-miscaptures.py: handle destructuring function arguments correctly. refs #1555

[david-sarah@…]

In [5441/ticket999-S3-backend]:

check-miscaptures.py: Python doesn't really have declarations; report the topmost assignment. refs #1555

[david-sarah@…]

In [5442/ticket999-S3-backend]:

check-miscaptures.py: handle corner cases around default arguments correctly. Also make a minor optimization when there are no assigned variables to consider. refs #1555

[david-sarah@…]

In [5443/ticket999-S3-backend]:

check-miscaptures.py: report the number of files that were not analysed due to syntax errors (and don't count them in the number of suspicious captures). refs #1555

[davidsarah]

The current version also detects captures of variables declared or modified in list comprehensions and while loops.

[david-sarah@…]

In be1c94893fa08ec8:

Add misc/coding_tools/check-miscaptures.py to detect incorrect captures of variables declared in a for loop, and a 'make check-miscaptures' Makefile target to run it. (It is also run by 'make code-checks'.) This is a rewritten version that reports much fewer false positives, by determining captured variables more accurately. fixes #1555

[david-sarah@…]

In d2f3ef9714f4db2d:

check-miscaptures.py: check while loops and list comprehensions as well as for loops. Also fix a pyflakes warning. refs #1555

[david-sarah@…]

In 1c6fe1d23080396d:

check-miscaptures.py: handle destructuring function arguments correctly. refs #1555

[david-sarah@…]

In 5359c24e9901c588:

check-miscaptures.py: Python doesn't really have declarations; report the topmost assignment. refs #1555

[david-sarah@…]

In efd3c8b11361913c:

check-miscaptures.py: handle corner cases around default arguments correctly. Also make a minor optimization when there are no assigned variables to consider. refs #1555

[david-sarah@…]

In 4d6260d6ad5f5400:

check-miscaptures.py: report the number of files that were not analysed due to syntax errors (and don't count them in the number of suspicious captures). refs #1555