[tahoe-lafs-trac-stream] [tahoe-lafs] #591: "make quicktest" could be quicker and less noisy

[tahoe-lafs]

#591: "make quicktest" could be quicker and less noisy
---------------------------+------------------------------
     Reporter:  warner     |      Owner:  warner
         Type:  defect     |     Status:  closed
     Priority:  major      |  Milestone:  undecided
    Component:  packaging  |    Version:  1.2.0
   Resolution:  fixed      |   Keywords:  test performance
Launchpad Bug:             |
---------------------------+------------------------------
Changes (by daira):

 * status:  new => closed
 * resolution:   => fixed


Old description:

> I added the "quicktest" target to the Makefile long ago, just after Zooko
> added some dependencies to make sure that "make test" would cause a "make
> build" to occur first. This dependency reduces the surprise factor for a
> new developer who grabs a source tree and types "make test" without first
> typing "make". I don't mind that sort of handholding, but I wanted a way
> to tell the system that I know what I'm doing and I really do want to run
> just the tests and not do anything else. The "quicktest" target was added
> to run trial without the extra fluff. In addition, I don't want to be
> distracted by a lot of extra output.
>
> In the recent setup.py refactoring (specifically the part that uses a
> setuptools plugin to invoke trial, instead of custom code in setup.py),
> the "quicktest" target is starting to look more like the not-so-quick
> "test" target.
>
> On my workstation, the tahoe-1.2.0 release could do a "make quicktest
> TEST=allmydata.test.test_base62" in 1.0 seconds, and emits 17 lines
> (only one of which is not coming from trial, the line which tells you
> which 'trial' command line you would need to run the tests yourself). The
> underlying trial process reports a runtime of about 0.1 seconds. The
> corresponding "make test TEST=allmydata.test.test_base62" process takes
> 3.5s and emits 159 lines (so 142 lines of fluff). If I wanted more
> control over the trial command, I could cut/edit/paste the command line
> it presented.
>
> In current trunk, the same quicktest command takes 2.5s and emits 99
> lines (82 lines of fluff). The non-quick "test" command takes 3.6s and
> emits 177 lines (160 lines of fluff). In addition, the trial command line
> is no longer available, and I'm not even sure it is easy to figure out
> what the user might be able to run to invoke the same tests without the
> setup.py framework.
>
> It's disappointing that the "quicktest" target is becoming just as slow
> and just as noisy as the "test" target. I don't mind the extra couple of
> seconds.. if it were 5 seconds, that would cut into my edit-test-repeat
> loop (by giving me time to get distracted while waiting for a single-test
> development cycle to complete), but 2.5s is not quite enough to hit that
> threshold. However, the extra 82 lines of fluff is annoying: when I tell
> emacs to re-run the unit test for whatever feature I'm working on right
> now, the test results no longer fit in the window that it creates.
>
> So, it's low-priority, but I'd appreciate it if our ongoing setup.py work
> would consider brevity and speed of testing to be a priority, in
> particular the use-case of running a single test case (in an edit-test-
> repeat cycle).

New description:

 I added the "quicktest" target to the Makefile long ago, just after Zooko
 added some dependencies to make sure that "make test" would cause a "make
 build" to occur first. This dependency reduces the surprise factor for a
 new developer who grabs a source tree and types "make test" without first
 typing "make". I don't mind that sort of handholding, but I wanted a way
 to tell the system that I know what I'm doing and I really do want to run
 just the tests and not do anything else. The "quicktest" target was added
 to run trial without the extra fluff. In addition, I don't want to be
 distracted by a lot of extra output.

 In the recent setup.py refactoring (specifically the part that uses a
 setuptools plugin to invoke trial, instead of custom code in setup.py),
 the "quicktest" target is starting to look more like the not-so-quick
 "test" target.

 On my workstation, the tahoe-1.2.0 release could do a "make quicktest
 TEST=allmydata.test.test_base62" in 1.0 seconds, and emits 17 lines  (only
 one of which is not coming from trial, the line which tells you which
 'trial' command line you would need to run the tests yourself). The
 underlying trial process reports a runtime of about 0.1 seconds. The
 corresponding "make test TEST=allmydata.test.test_base62" process takes
 3.5s and emits 159 lines (so 142 lines of fluff). If I wanted more control
 over the trial command, I could cut/edit/paste the command line it
 presented.

 In current trunk, the same quicktest command takes 2.5s and emits 99 lines
 (82 lines of fluff). The non-quick "test" command takes 3.6s and emits 177
 lines (160 lines of fluff). In addition, the trial command line is no
 longer available, and I'm not even sure it is easy to figure out what the
 user might be able to run to invoke the same tests without the setup.py
 framework.

 It's disappointing that the "quicktest" target is becoming just as slow
 and just as noisy as the "test" target. I don't mind the extra couple of
 seconds.. if it were 5 seconds, that would cut into my edit-test-repeat
 loop (by giving me time to get distracted while waiting for a single-test
 development cycle to complete), but 2.5s is not quite enough to hit that
 threshold. However, the extra 82 lines of fluff is annoying: when I tell
 emacs to re-run the unit test for whatever feature I'm working on right
 now, the test results no longer fit in the window that it creates.

 So, it's low-priority, but I'd appreciate it if our ongoing setup.py work
 would consider brevity and speed of testing to be a priority, in
 particular the use-case of running a single test case (in an edit-test-
 repeat cycle).

--

Comment:

 I think #1296 can be considered to have fixed this ticket.

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/591#comment:18>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage