Testing my website for visual regressions with Playwright snapshot tests

Making changes to websites is tricky, because even small changes may lead to visual regressions in the page layout. For example, changing a bit of CSS to fix something somewhere may blow up in a completely different place. I could just click around and make sure everything looks okay, but that is boring and error-prone. What if I told you that there is a way to make sure that every pixel on your site is perfect, staying just they way you intended it to be?

It’s called Playwright snapshot testing (or “visual comparisons”). Playwright can be used to take screenshots of the whole website, and then compare it to older screenshots. This way, no unintended changes can slip away into production. You can even test Chromium, Firefox and Webkit (Safari) at the same time!

With Playwright, I can run a command (pnpm run test-ui) to open a new Playwright window from which I can run the tests each time I make a change.

I can do this for every page on my website, in parallel, in about twelve seconds. And with Playwright, it’s really easy to do!

How-to #

Assuming you have a package.json file and Playwright installed, open up playwright.config.ts and change baseURL to your development server URL:

{
	// ..
	use: {
	    // Base URL to use in actions like `await page.goto('/')`
	    baseURL: 'http://127.0.0.1:1313',
	    // ..
	},
}

In my case, I used Hugo’s default development server port 1313.

You can delete all other tests in the tests/ directory and create a single visual-testing.spec.ts file with the following code:

import { test, expect } from '@playwright/test'

const paths = ['/', '/about', '/projects', '/posts']

test.describe.configure({ mode: 'parallel' })

for (const path of paths) {
  test(path, async ({ page }) => {
    await page.goto(path)

    await expect(page).toHaveScreenshot({
      fullPage: true,
      // Apply mask over all images.
      // Some images, like GIFs, can change between snapshots
      // and lead to flaky tests.
      mask: await page.getByRole('img').all(),
    })
  })
}

If you only have a handful of pages, that’s it! You can open the UI with pnpm exec playwright test --ui or update the snapshots with pnpm exec playwright test --update-snapshots. The first time you run the tests it will look like all of them were cancelled, but this happens because none of the pages have snapshots yet. After the second run, all tests should appear green.

Since I want to also check every post I have published, I’ve written a bit more code to get the paths of every post before the tests run. Here is my full code:

import { test, expect } from '@playwright/test'

import matter from 'gray-matter'
import * as fs from 'node:fs'
import { globSync } from 'glob'
import path from 'node:path'

let files = globSync(['content/posts/**/*.md'])

let paths = files.flatMap(file => {
  // Some posts are content/posts/{slug}/index.md
  // others are content/posts/{slug}.md
  // Extract {slug}.
  let slug = (path.basename(file) === 'index.md')
    ? path.dirname(file).slice("content/".length)
    : `posts/${path.basename(file, '.md')}`

  let frontmatter = matter(fs.readFileSync(file))
  return frontmatter.data.draft ? [] : [slug]
})

paths.push('/', '/about', '/projects', '/posts', '/404')

test.describe.configure({ mode: 'parallel' })

for (const path of paths) {
  test(path, async ({ page }) => {
    await page.goto(path)

    await expect(page).toHaveScreenshot({
      fullPage: true,
      // Apply mask over all images.
      // Some images, like GIFs, can change between snapshots
      // and lead to flaky tests.
      mask: await page.getByRole('img').all(),
    })
  })
}

I also have an entry in the playwright.config.ts file to spin up Hugo before running any tests. It’s faster to run things locally, but you could

{
	// ...
	webServer: {
	    command: 'hugo serve',
	    url: 'http://127.0.0.1:1313',
	    reuseExistingServer: !process.env.CI,
	},
}

The only issue I have with Playwright is that it doesn’t support Arch Linux on my laptop (only Windows, MacOS and Ubuntu).