Nightwatch.js Blog

← Back

Testing WebRTC Apps with Nightwatch

Overview of WebRTC

WebRTC is a set of standards that enables realtime, peer to peer audio, video and data streaming between browser clients without any plug-ins. Its main uses today are audio/video conferencing, screen-sharing apps and multiplayer games but it can have other uses as well, like interacting with more tradidional SIP endpoints.

If you're new to WebRTC there are some great introductory articles at HTML5Rocks and Mozilla Developer Network. WebRTC is now available in Chrome, Firefox and Opera and there are already plenty of apps and websites enabling users to communicate using this relatively new technology.

Getting started

Automated testing of WebRTC applications can be cumbersome since it involves not only starting fake media devices but also starting more than one browser. A reliable test should check if at least two clients are connected and that they can interact with each other.

All that probably sounds like quite a bit of work but actually it's not that difficult to get it all working with Nightwatch. As a reference we'll use the article that Philipp Hancke at andyet.com has wrote for his blog. They run a quality web conferencing service at talky.io and have a demo webrtc service at simplewebrtc.com/demo.html which we'll use in our test.

We'll start by creating a typical Nightwatch-enabled project and downloading all the needed binaries, like the selenium server and the chrome driver and installing Nigthwatch (with npm install -g nightwatch) if it's not already installed.

Then will create our nightwatch.json configuration file, which will look like this:

  
{
  "src_folders" : ["./tests"],
  "output_folder" : "./reports",
  "live_output" : true,
  "parallel_process_delay" : 1500,

  "selenium" : {
    "start_process" : true,
    "server_path" : "./bin/selenium-server-standalone-2.43.1.jar",
    "log_path" : "",
    "cli_args" : {
      "webdriver.chrome.driver" : "bin/chromedriver"
    }
  },

  "test_settings" : {
    "default" : {
      "launch_url" : "https://simplewebrtc.com/demo.html",
      "silent" : true,
      "screenshots" : {
        "enabled" : false,
        "path" : "./screenshots"
      },
    },

    "chrome" : {
      "desiredCapabilities" : {
        "browserName" : "chrome",
        "chromeOptions" : {
          "args" : [
            "use-fake-device-for-media-stream", 
            "use-fake-ui-for-media-stream"
          ]
        }
      }
    }
  }
}

For this prototype we'll be using chrome since it is easier to simulate a fake media device than Firefox. I have created two test environments, a default one and another named chrome.

Using fake media device

Chrome provides two handy cli flags which we can use in our project to make things easier for testing:

  • --use-fake-device-for-media-stream - simulates a fake webcam and mic for testing
  • --use-fake-ui-for-media-stream - allows skipping the security prompt for sharing the media device

Test overview

Next we'll proceed with writing the actual test. Currently the test project looks like this:

├── bin/
|   ├── selenium-server-standalone-2.43.1.jar
|   └── chromedriver
├── lib/  
|   └── custom-commands/
├── reports/
├── screenshots/  
├── tests/  
├── nightwatch.json
└── selenium-debug.log
The test will do the following:
  • open two browsers in parallel and navigate to the given url - the second one will open with a slight delay
  • wait a number of seconds to become connected
  • check if both video streams are connected
  • when the second client is closed, check if the remote video is removed from the page

Running tests in parallel

We'll make use of one of the relatively new features in Nightwatch and still in an experimental stage, the ability to run the tests in parallel.

If you're not familiar with this feature, the way it works is: say you have a few environments created in your nightwatch.json file under test_settings. You have a default environment, another one for chrome and another one for firefox. If you want to run the tests against both chrome and firefox environments in parallel, for instance, you simply specify both of them in the command line, like so:

$ nightwatch --env chrome,firefox

The test runner will then start each of them in a separate child_process. You can read a bit more about this in the Developer Guide.

Starting more than one client

Now the fun part begins. To start a second client we need to use the above feature, i.e. running the test in parallel. But simply running the same test in two browsers in parallel is not enough. We'll do the testing only for the first client and the second one will just wait a few seconds and then exit.

To start two clients will just specify the chrome environment twice in the command line:

$ nightwatch --env chrome,chrome

We could also start a firefox environment but using a fake media device in Firefox is not very straightforward so we'll pass on that for now.

Adapting test settings at run-time per environment

By default both clients will run the test the same way and output the results to stdout and write a JUnit XML report. Since the browser is the same the report file name will be the same as well and that means the second client's report will overwrite the first one. That's not very good because most of the testing will be done for the first client.

The way we fix this is by disabling the output at run-time for the second environment. And we'll do this in the nightwatch.conf.js configuration file, which looks a bit like this:

module.exports = (function(settings) {
  if (process.env.__NIGHTWATCH_ENV_KEY !== 'chrome_1') {
    settings.output_folder = false;
    settings.output = false;
  }
  return settings;

})(require('./nightwatch.json'));

The nightwatch.conf.js always takes precedence over nightwatch.json if both are present.

In the above we're reading the system environment variable __NIGHTWATCH_ENV_KEY which the Nightwatch runner populates with the current value of the testing environment that is being used - in this case chrome followed by a 1-based index: chrome_1. We want to disable the output for all other environments so we wont have any report file clashes.

We'll be using the same way of distinguishing between testing environments in the actual test.

Writing the actual test

It is important to be able to know inside the test which client (i.e. environment) it is the current one so that it performs different actions. We'll be reading the same system variable as in the previous step: __NIGHTWATCH_ENV_KEY.

The test will do the following:

  • open up the given url and wait for body element to be visible
  • first client only:
    • wait for #localVideo element to become connected
    • wait for remote video element to become connected
    • after the remote video is connected, waits 1000ms
    • wait for the remote video element to be removed - i.e. the other peer has left the room

  • second client only - no assertions performed, only pauses for 10 seconds and then exits


module.exports = new (function() {  
  var firstClient = process.env.__NIGHTWATCH_ENV_KEY == 'chrome_1';
  var testCases = this;

  testCases['opening the browser and navigating to the url'] = function (client) {
    client
      .url('https://simplewebrtc.com/demo.html?nightwatchjs')
      .waitForElementVisible('body', 1000);
  };

  if (firstClient) {
    testCases['wait for clients to become connected'] = function(client) {
      client
        .waitForElementVisible('#localVideo', 1500)
        .waitForClientConnected('#localVideo', 5000)
        .waitForClientConnected('#remotes .videoContainer:nth-child(1) video', 8000,
          'Remote video stream (%s) was connected in %s ms.');
    };

    testCases['wait for peer to disconnect'] = function (client) {
      client
        .pause(1000)
        .waitForElementNotPresent('#remotes video', 10000);
    };
  } else {
    testCases.suspend = function (client) {
      client.pause(10000);
    };
  }

  testCases.after = function(client) {
    client.end();
  };

})();

Checking if the clients are connected

You may have noticed in the above code snippet the presence of a custom command: waitForClientConnected. The way this works is it checks periodically for the readyState property of the supplied element to equal 4 (HAVE_ENOUGH_DATA).

The possible values of the readyState property are as follows:

// ready state
const unsigned short HAVE_NOTHING = 0;  
const unsigned short HAVE_METADATA = 1;  
const unsigned short HAVE_CURRENT_DATA = 2;  
const unsigned short HAVE_FUTURE_DATA = 3;  
const unsigned short HAVE_ENOUGH_DATA = 4;  
readonly attribute unsigned short readyState;  

More information is available on the W3C Video Element page.

Resources

That's about it. The test project is available on GitHub.

Read more about WebRTC here: