#1001 new defect

test framework for web apps that use JavaScript

Reported by: davidsarah Owned by: davidsarah
Priority: major Milestone: undecided
Component: code Version: 1.6.0
Keywords: jsui music test Cc:
Launchpad Bug:

Description

If we add dependencies on JavaScript to Tahoe, such as the 'Tahoe Explorer' frontend as proposed in #1000, we will need a test framework for the JS code. Ideally, it should be able to test interactions between JavaScript and Python, and automatically run tests in multiple web browsers.

Attachments (6)

test_explorer.py (948 bytes) - added by davidsarah at 2010-04-13T02:36:58Z.
Dummy test that opens Firefox, IE, Safari and Chrome using Windmill
tilting.py (2.1 KB) - added by davidsarah at 2010-04-13T02:54:31Z.
Simple hack to run Windmill tests from a trial test suite
tilting.2.py (9 bytes) - added by davidsarah at 2010-04-13T02:56:35Z.
ignore me (file deleted)
add-windmill-dependency-darcspatch.txt (1.8 KB) - added by davidsarah at 2010-04-16T19:16:10Z.
Add dependency on Windmill (fixed patch)
tilting3.py (6.3 KB) - added by josipl at 2010-05-13T19:56:46Z.
test_dummy.py (515 bytes) - added by josipl at 2010-05-13T19:58:17Z.
Dummy test which should run JS tests and open Firefox.

Download all attachments as: .zip

Change History (20)

comment:1 Changed at 2010-03-17T03:59:00Z by zooko

I looked into what node.js uses since I consider node.js to be inspired by Twisted (at least by Twisted's event-based concurrency) and Twisted has an excellent tradition of testing. Here's what I found: http://debuggable.com/posts/unit-testing-with-node-js:4b647d40-34e4-435a-a880-3b04cbdd56cb

comment:2 Changed at 2010-04-13T02:22:59Z by davidsarah

  • Milestone changed from undecided to 1.8.0
  • Owner changed from somebody to davidsarah

I've been investigating Windmill, which seems to have similar functionality to Selenium in terms of web browser testing, but which uses Python as its scripting language (in addition to JavaScript). It looks like a very good fit, because we need to be able to script interactions between Python code and a web app (for example, setting up a temporary grid + directory structure in Python and then testing the web app against it, or hooking Python objects to make sure they are receiving the intended requests from the app).

I was initially concerned that Windmill might be overkill. However, if we need to be able to test Tahoe Explorer (#1000) as well as Josip Lisec's Music player and Hasini Gunasinghe's Blog for GSoC (if those projects are accepted), then I think we may end up needing a fair amount of the functionality that it provides over and above simpler JS test frameworks like JSUnit or node.js.

Windmill is Apache 2.0-licensed.

comment:3 Changed at 2010-04-13T02:31:09Z by davidsarah

  • Summary changed from test framework for JavaScript to test framework for web apps that use JavaScript

Changed at 2010-04-13T02:36:58Z by davidsarah

Dummy test that opens Firefox, IE, Safari and Chrome using Windmill

Changed at 2010-04-13T02:54:31Z by davidsarah

Simple hack to run Windmill tests from a trial test suite

Changed at 2010-04-13T02:56:35Z by davidsarah

ignore me (file deleted)

comment:4 Changed at 2010-04-14T03:01:19Z by davidsarah

  • Keywords music added

comment:5 follow-up: Changed at 2010-04-14T14:06:07Z by josipl

I agree with David-Sarah that Windmill, in the long run, is better option.

It probably is a bit overkill for simpler tests, but assertion module from node.js just can't cover all of the functionality - with node.js one can't test the actual UI part since there is no DOM API provided by default. Module which provides DOM API exists but is far from complete and it has dependencies on other modules/tools, and thus I'm not sure if introducing all these dependencies is worth the trouble.

On the other hand, Windmill runs the tests directly in the browser and thus allows testing every aspect of app's functionality. Also, Windmill's IDE seems like an rather excellent tool.

comment:6 in reply to: ↑ 5 Changed at 2010-04-15T18:21:05Z by davidsarah

  • Keywords review-needed added

Replying to josipl:

... with node.js one can't test the actual UI part since there is no DOM API provided by default. Module which provides DOM API exists but is far from complete and it has dependencies on other modules/tools, and thus I'm not sure if introducing all these dependencies is worth the trouble.

Also, that would only be testing against that particular implementation of the DOM, rather than the implementations used by Firefox, IE, Safari, and Chrome.

I'm going to attach a patch that adds a dependency on Windmill to source:_auto_deps.py . Please test that this doesn't break anything on any platform (it doesn't on win32), but don't commit to trunk yet.

comment:7 Changed at 2010-04-15T18:33:12Z by davidsarah

Oh, please test that after applying this patch and installing Tahoe using python setup.py install, that python -c "import windmill; help(windmill)" works. It should print a package description.

Changed at 2010-04-16T19:16:10Z by davidsarah

Add dependency on Windmill (fixed patch)

comment:8 Changed at 2010-04-23T23:58:13Z by davidsarah

  • Keywords review-needed removed

The patch has been applied, and hasn't caused any build failures so far (but not all of the buildslaves are on-line at the moment).

comment:9 Changed at 2010-05-04T10:14:06Z by zooko

Now that more builders are working, we can see that this patch breaks the "Desert Island" test:

http://tahoe-lafs.org/buildbot/builders/clean/builds/2262/steps/test-desert-island/logs/stdio

This build step ends with:

python misc/check-build.py make.out no-downloads

Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Reading http://pypi.python.org/simple/windmill/
Reading http://www.getwindmill.com/
Reading http://windmill.osafoundation.org/
Downloading http://pypi.python.org/packages/source/w/windmill/windmill-1.3.tar.gz#md5=1f5a8fca648beeaacbfc8f0e01d41c8c
Failed: build tried to download files

To make it pass this test, add a copy of windmill into the file that it fetches at the beginning of the test:

test -f tahoe-deps.tar.gz || wget http://allmydata.org/source/tahoe/deps/tahoe-deps.tar.gz

comment:10 follow-up: Changed at 2010-05-04T16:14:14Z by davidsarah

Since we don't actually use Windmill yet, and it's 3.2 MiB, I think we should just comment out the dependency for 1.7. We've already tested that it installs on all supported platforms, which was the intention of applying the patch.

comment:11 in reply to: ↑ 10 Changed at 2010-05-04T17:57:22Z by davidsarah

Replying to davidsarah:

Since we don't actually use Windmill yet, and it's 3.2 MiB, I think we should just comment out the dependency for 1.7.

Done in 43b092ffeb7cae9e.

comment:12 Changed at 2010-05-13T19:55:21Z by josipl

I'm attaching a modified version of tilting.py which was discussed the other day in IRC along with another dummy test.

TiltingMixin's setUp method now starts an introducer and storage nodes. Results of JavaScript tests are being reported in a manner which test suite can easier pickup.

test_dummy.py file contains a dummy test which copies code of an imaginary application to public_html directory of a started storage node and then starts JavaScript tests.

There are some unresolved issues with code: I couldn't find a way to disable Windmill from printing to stdout (someone on Windmill's IRC channel suggested that it should be simply used as a Nose plugin). The other problem, is the way waiting for introducer.furl file is implemented - using a loop and sleep(), this will be fixed once Python API for creating/starting/stopping nodes gets implemented.

Changed at 2010-05-13T19:56:46Z by josipl

Changed at 2010-05-13T19:58:17Z by josipl

Dummy test which should run JS tests and open Firefox.

comment:13 Changed at 2010-06-09T04:49:34Z by zooko

Note that such a dependency should probably be added to {{tests_require}}} in setup.py and not to install_requires in _auto_deps.py.

comment:14 Changed at 2010-08-14T06:45:30Z by zooko

  • Milestone changed from 1.8.0 to undecided
Note: See TracTickets for help on using tickets.