How Promises and Tasks are Improving Tests

March 30, 2014 at 02:15 PM | categories: Mozilla, JavaScript

I was a very early adoptor of promises and Tasks in Firefox's JavaScript code base. To me, promises on their own are ok. The ability to chain promises together and tack one error handler on the end sure beats the Pyramid of Doom and having to pass errors into callbacks everywhere. But what really lured me in were tasks: using generators (then a feature only available in SpiderMonkey) to represent async code flow as nice, easy-to-read procedural flow that nearly every programming can relate to. It made code much easier to read and grok. I've been using tasks ever since.

When I started writing new APIs that returned promises instead of using callbacks, I found myself writing a lot of tests consuming promises and using tasks. So, I added an add_task API to our xpcshell test harness to make writing task-based unit tests involve less boilerplate. That API is now used heavily for new xpcshell tests.

While I initially added add_task() to cut down on the boilerplate for writing tests, I only recently realized it has another benefit: it's helped cut down on hung tests!

Before, with callback-based APIs, we'd code tests like so:

add_test(function () {
  do_something(function onThatThing(result) {
     Assert.ok(result.success);
     run_next_test();
  });
});

Or another pattern:

add_test(function () {
  do_something(function onThatThing(result) {
    // The next line throws an Error by accident!
    result.foo();
    run_next_test();
  });
});

In the first example, the test will hang if the callback never gets called. The test harness driver will eventually terminate the test (after a multi-second delay with no output). Not good.

In the second example, we are still susceptible to the callback not being called. But we have a different problem: an untrapped Error is thrown from a callback! This results in the same behavior: run_next_test() (the function that says to advance to the next test) won't execute and the test will hang until it times out.

A more proper way to write this test is:

add_test(function () {
  do_something(function onThatThing(result) {
    try {
      result.foo();
    } catch (ex) {
      do_report_unexpected_exception(ex);
    }

    run_next_test();
  });
});

In reality, few people surround all their callbacks with try..catch blocks because, well, it's a lot of typing and people don't always think it's necessary (the test passes most of the time, doesn't it?).

What promises and task-based tests are doing is enabling us to write more robust tests without all of the extra work. Here is how you would use task-based tests:

add_task(function* () {
  let result = yield do_something();
  // The next line throws an Error by accident!
  result.foo();
});

Here, the Error thrown by the test function is thrown within the context of an executing Task. It is caught by the Task and converted into a rejected promise. The test harness sees that failure immediately and no timeout occurs! This can cut down on overhead when writing tests, especially if you are trying to debug a hang.

Furthermore, the test is 4 lines versus 10. Less typing means you have more time to write additional tests or you can focus on writing other patches.

Finally, the task-based test functions are easier to understand. That 4 line, procedural test is much easier to grok than its callback-based counterpart.

And before I conclude, I should mention that we can do more with promises. For example, bug 976205 is making uncaught promise errors turn into test failures! There is also an awesome patch in bug 867742 to introduce a unified JavaScript test harness API for defining JavaScript tests in our tree (currently the APIs for xpcshell tests and mochitests are different, leading to cognitive dissonance and lower productivity). If you want to be a hero to the Firefox developer community, help finish that patch.

Given that so much Firefox feature development time (at Mozilla) is spent writing and debugging tests, I encourage everyone to consider promises and tasks for his or her next feature so that you can cut down on development time and complete projects faster.