Calling Software Test Engineers! Is it time to replace Nightwatch with Playwright?

By Aska Cloke

We spare you the pain of assessing these two automation tools by looking at how they compare to each other

I have worked with various automation tools throughout my career — starting with QuickTest Professional (QTP), now called Unified Functional Testing (UFT), and, most recently, Playwright. One of my least favourite tools in the past was Nightwatch. However, it seems that Nightwatch has improved significantly, so I thought I'd give it another look and see how it stands up to Playwright today!

This article is for Software Test Engineers. I hope I can spare you some pain in comparing tools and maybe even share some practical knowledge about these modern testing frameworks. Empowered with that knowledge, we can all make informed decisions when choosing a testing framework.

Built-in functionalities

When I first used Nightwatch back in 2016/2017, I don’t recall any of the below functionalities being built-in. Instead, we had to code them explicitly. Fast forward to 2024, and these built-in features are now available in both Nightwatch and Playwright.

1. Auto wait and auto retries

Both Nightwatch and Playwright automatically wait for elements to appear before attempting to interact with them. For example, if your test requires you to enter a username into an input box but the screen takes time to load, both tools will keep checking for the input box to be available until they time out. As a result, you can remove the 'wait until' code for potentially slow-loading pages, making the tests clearer and easier to read. The default timeout setting is five seconds for Nightwatch and 30 seconds for Playwright, but this can be reconfigured in the configuration file.

The automatic retry feature also simplifies the test script. When an assertion fails, it automatically re-asserts the failed line a specified number of times, which can be configured in the configuration file.

2. Handling of pop-up messages

Both Nightwatch and Playwright can handle pop-up messages by explicitly waiting for them. However, Playwright takes it a step further by making it easier to handle pesky, unpredictable pop-ups that appear at different times using page.addLocateHandler(). When an overlay is detected at any point during the test, Playwright first calls the handler, and then proceeds with the specified action to remove the overlay. This feature is unique to Playwright.

test('pop-up test', async({page}) => {
    await page.goto('https://www.techlistic.com/p/selenium-practice-form.html');
    await page.addLocatorHandler(
      page.getByRole('heading', { name: 'Privacy & Transparency' }),
      async () => {
          // This line is executed whenever overlay is detected.
        await page.click('#ez-accept-necessary');  
      });

      await expect(page.getByRole('heading', { name: 'Demo Automation Practice Form' })).toBeVisible();
      await page.locator('input[name="firstname"]').fill('test-firstname');
  })

Playwright addLocateHander() example

3. Reporting tool

Both of these automation tools come with stylish HTML reports and the frameworks add screenshots when a test fails. However, I found Nightwatch's report easier to navigate, as everything can be viewed on one screen. With Playwright, you have to navigate back and forth to check individual test results.

Nightwatch HTML report

Playwright HTML report

You have to open the individual test to see the details

4. Screenshots

Both Nightwatch and Playwright can take screenshots explicitly in code. Also, via a config file, you can enable automatic screenshots for failing tests.

Nightwatch

Test file

browser.saveScreenshot('./screenshots/screenshot.png')

nightwatch.conf.js

test_settings: {
    default: {
      screenshots: {
        enabled: true,
        path: 'screens',
        on_failure: true
      },

Playwright

Test file

await page.screenshot({ path: 'screenshot.png' });

playwright.config.js

use: {
    screenshot: 'only-on-failure',
  },

5. Test recorder/code generator

Nightwatch uses a Chrome extension that can automatically generate tests by recording user actions. From the DevTools, you can record multiple tests and export them to Nightwatch. While it can be a bit clunky to use, it allows you to record actions and add assertions. However, as seen in the exported script, the locators it captures are not very optimised (we should always choose ‘asserting by role’ over ‘selector’ to test accessibility).

Chrome recording

Exported script

describe("nightwatch", function () {
it("tests nightwatch", function (browser) {
  browser.windowRect({width: 1705, height: 693})
  .navigateTo("https://www.nearform.com/")
  .click("ul.nav_right > li:nth-of-type(2) > a")
  .click("#firstname-13c6a12c-2dbc-4455-a6cc-694b8bf39cb8")
  .setValue("#firstname-13c6a12c-2dbc-4455-a6cc-694b8bf39cb8", "test")
  .click("div.hs_submit input")

        .waitForElementVisible("text/Please complete all", function(result) {
          if (result.value) {
            undefined
          }
        })

  .end();
  });
});

In comparison, Playwright has its own test recorder called codegen, which can be launched by running the below code:

npx playwright codegen

Test code is generated in real-time as you navigate through the page. It is intuitive and quite user-friendly. You can record actions, capture page locators, and add assertions. Additionally, it supports device emulation and allows you to specify geolocation, language, and time zone. The power of this feature is hard to describe. Check out all of the features here.

VSCode extension

Running and scripting tests gets a whole lot easier with a VSCode extension. Both Nightwatch and Playwright have extensions, but they have slightly different functionalities. Let’s compare them.

Nightwatch extension

Test Explorer

• Nightwatch tests integrate directly into VSCode's built-in 'Test Explorer' into a dedicated section. You can execute tests from this section or directly from a test file.

Browsers

• Choose ‘Browser’ from the ‘Nightwatch Environment’ section.

• You can only select one browser at a time from this section.

Settings

• Select ‘Headless’ or ‘Headed’ mode.

• Open report automatically after tests have run.

• Set the number of parallels.

Playwright extension

Test Explorer

• All executable Playwright tests appear in this section.

Browsers

• You can select multiple browsers.

Settings

• Select ‘Headless’ or ‘Headed’ mode.

• Launch ‘Trace Viewer’, which enables you to follow tests step by step along with screenshots and information you can get from a browser dev tool, such as network trace. This makes debugging easier.

Tools

• One feature that sets Playwright apart from Nightwatch is its tool for selecting locators. I found the 'Pick Locator' feature to be very handy. You can point to the element you need, and the page locator is captured into VSCode. By pressing the Enter key on the captured locator, it is copied and is then ready to be pasted into your code.

• The ‘Record New’ functionality is almost exactly the same as codegen. The only difference is that it creates a new test file, then the test code is generated there as you navigate through the screen.

I rarely use codegen because I can do most jobs from the VSCode extension tools. If you haven’t used them before, I highly recommend giving them a try.

Language support, syntax comparison and debugging

Below, we look at language support, syntax and debugging for Nightwatch and Playwright.

1. Language support

Nightwatch supports JavaScript and TypeScript, making it ideal for teams working within a JavaScript ecosystem. Playwright, however, offers broader language support, including Java, Python, and .NET, in addition to JavaScript and TypeScript.

2. Syntax comparison

Up until v2, Nightwatch uses a traditional callback or promise style. Each command, like .findByRole or .url, relies on a callback function that contains assertions or further actions. WebDriver is used to execute tests.

*However, v3 (July 2023) introduced synchronous syntax.

Nightwatch test — up to v2

module.exports = {
    'nightwatch - assert filter and title': function(browser){
        browser.url(browser.launch_url);
        browser
            .element.findByRole('button', {name: 'Reject All'}).click();
        browser
            .assert.visible('input[placeholder="Filter any column..."]')
            .assert.titleEquals('Demo - Performance Grid');
    }
}

Nightwatch using async/await (from v3)

module.exports = {
    'nightwatch - assert filter and title': async function(browser) {
    await browser.url(browser.launch_url);
    await browser.element.findByRole('button', { name: 'Reject All' }).click();
    await browser.assert.visible('input[placeholder="Filter any column..."]');
    await browser.assert.titleEquals('Demo - Performance Grid');
    }
};

Playwright uses async/await syntax and tests are executed directly in the browser.

test('playwright - assert filter and title', async ({ page }) => {
    await page.goto('/example');
    await page.getByRole('button', { name: 'Reject All' }).click();
    // break for debug here
    await expect(page.getByPlaceholder('Filter any column...')).toBeVisible();
    await expect(page).toHaveTitle('Demo - Performance Grid')
  });

3. Debugging

Previously, Nightwatch only supported callback-based syntax, which made debugging more challenging. However, this was addressed in mid-2023, making debugging simpler and allowing for a more meaningful comparison between Nightwatch and Playwright. Let’s compare what happens when you try to debug the above sample code.

1. Put in a break line, then click the ‘Reject All’ button.

2. Execute tests in debug mode.

3. The test reaches the break line.

Conclusion

Nightwatch has significantly improved over the years, and I am impressed by how user-friendly it has become. However, I still prefer Playwright over Nightwatch due to some limitations that are difficult to overcome. For instance, Nightwatch requires the setup and maintenance of WebDriver. Up until mid-2023, Nightwatch only supported callback-based syntax, which made debugging more challenging. Ease of debugging is crucial when determining the cause of a test failure, as you need to quickly ascertain whether it is a valid failure or an error in the test script. While Nightwatch now supports async/await syntax, it came a bit too late, as Playwright has been using this feature since 2020.

Playwright also supports more languages, making it adaptable to a wider range of development stacks. The framework is easier to maintain since there is no need for a separate WebDriver, and tests are executed directly in the web browser. Its code is simpler to script and read, which makes the test development process more efficient. Thus, I’ll be sticking with Playwright until another new tool emerges for evaluation.