[tahoe-lafs-trac-stream] [tahoe-lafs] #1555: add check-miscaptures.py script to check for incorrect variable captures in for loops

[tahoe-lafs]

#1555: add check-miscaptures.py script to check for incorrect variable captures in
for loops
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:  somebody
  davidsarah             |     Status:  reopened
         Type:           |  Milestone:  1.10.0
  enhancement            |    Version:  1.9.0a2
     Priority:  major    |   Keywords:  miscapture coding-tools review-
    Component:  code     |  needed
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Old description:

> The attached script checks for errors in which a variable declared in a
> {{{for}}} loop is captured by a lambda or function definition. If the
> lambda/function is executed after the loop iteration in which it was
> created, it will refer to some later value of the variable, which is
> usually not what was intended.
>
> The script currently produces false positives if a variable is declared
> and used in the same lambda/function within a for loop (i.e. it is not
> captured). I don't intend to fix that until after the Tahoe summit in
> November, probably. However, it produces relatively few false positives
> already, and found some genuine bugs (see #1556).

New description:

 The attached script checks for errors in which a variable declared in a
 {{{for}}} loop is captured by a lambda or function definition. If the
 lambda/function is executed after the loop iteration in which it was
 created, it will refer to some later value of the variable, which is
 usually not what was intended.

 ~~The script currently produces false positives if a variable is declared
 and used in the same lambda/function within a for loop (i.e. it is not
 captured). I don't intend to fix that until after the Tahoe summit in
 November, probably.~~ However, it produces relatively few false positives
 already, and found some genuine bugs (see #1556).

--

Comment (by davidsarah):

 The remaining significant false positive cases are where:
  * the capturing function is only used in the same iteration as the values
 it was ''intended'' to capture. For example, in
 {{{
 for x in xs:
     print map(lambda y: x+y, ys)
 }}}
    the use of each lambda is only in the same iteration as the {{{x}}} it
 was intended to capture (and {{{map}}} does not retain the function), so
 there is no bug.
  * the captured variables are only assigned once, before any application
 of a capturing function. A somewhat artificial example:
 {{{
 fs = []
 z = None
 for x in range(2):
     if z is None:
         z = 42
     fs.append(lambda y: y+z)
 }}}
    because the captured {{{z}}} is effectively constant, even though it is
 assigned in the loop.

 Both these cases are undecidable to detect in general. You could detect
 special cases of the first, such as builtin {{{map}}} or {{{filter}}} with
 a lambda expression as its first argument, but it's probably not worth the
 complication.

 After applying the patches on #1556, Tahoe produces no warnings from
 {{{make check-miscaptures}}}, so I'd like to include that in the buildbot
 flow in the same way as pyflakes. (It would be ok to run it in the same
 step as pyflakes.)

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1555#comment:7>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage