Visual regression testing with screenshots: Puppeteer, pixelmatch and what makes diffs flaky

Visual regression testing means taking a screenshot of a page, keeping it as the baseline, and comparing every later screenshot of the same page with it: if pixels change that should not, the test fails and shows you where. The idea is simple; making it reliable is not. The same page, shot twice a second apart, rarely gives identical pixels — and an image diff with default settings can miss a real change. Every number on this page was measured on 2026-11-07 with Puppeteer 25.7.0 (Chrome 152) in the official ghcr.io/puppeteer/puppeteer image and pixelmatch 6, on small local test pages.

For taking the screenshots themselves — viewport, element, retina, waiting — see the Puppeteer screenshot guide.

What we measured

Three lessons come out of that table. A page has to be stabilised before it can be compared. The baseline and the new shot have to come from the same environment — the same static page differed in 2,195 pixels between two official browser images, which ship different browser versions and fonts — on a page of plain text and one button. And the threshold is a colour tolerance per pixel, not a share of the image: at the defaults a whole button changing colour went unnoticed.

A baseline and a compare script

import fs from 'node:fs';
import puppeteer from 'puppeteer';
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';

const [url, name] = process.argv.slice(2);
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setViewport({ width: 1280, height: 800, deviceScaleFactor: 1 });
await page.goto(url, { waitUntil: 'networkidle2' });

// Stabilise: no animations, transitions or caret; fonts loaded.
await page.addStyleTag({ content: `*, *::before, *::after {
  animation: none !important; transition: none !important; caret-color: transparent !important; }` });
await page.evaluate(() => document.fonts.ready);

const shot = PNG.sync.read(await page.screenshot());
await browser.close();

const baselinePath = `baselines/${name}.png`;
if (!fs.existsSync(baselinePath)) {
  fs.writeFileSync(baselinePath, PNG.sync.write(shot));
  console.log('baseline written');
  process.exit(0);
}

const base = PNG.sync.read(fs.readFileSync(baselinePath));
if (base.width !== shot.width || base.height !== shot.height) {
  console.error(`size changed: ${base.width}×${base.height} → ${shot.width}×${shot.height}`);
  process.exit(1);
}

const diff = new PNG({ width: base.width, height: base.height });
const changed = pixelmatch(base.data, shot.data, diff.data, base.width, base.height, { threshold: 0 });
fs.writeFileSync(`diffs/${name}.png`, PNG.sync.write(diff));
console.log(`${changed} pixels differ`);
process.exit(changed > 0 ? 1 : 0);

Commit the baselines/ folder. When a change is intended, delete the baseline (or add an --update flag) and run again. The diff image paints changed pixels red over a faded copy of the page, which is what makes a failure readable in a pull request.

Check the size first. pixelmatch compares two buffers of the same size; it cannot compare a 1280-wide shot with a 1281-wide one, and a full-page screenshot changes height whenever content above the fold grows. A size change is a result in itself — report it, do not crop it away.

Stabilising the page

Each of these made a difference in our test or is a common cause of flaky diffs:

• Animations and transitions. A spinner rotates between shots. The CSS above stops them; for JavaScript animations, wait for the element to settle or hide it. Emulating prefers-reduced-motion only helps on sites that respect it.
• The text caret. A focused input blinks; caret-color: transparent removes it.
• Dates, times and random data. "Updated 2 minutes ago", today's date, a random avatar. Use fixed test data, or replace the text before the shot (page.evaluate), or cover the element.
• Fonts. Wait for document.fonts.ready; a web font arriving after the shot changes every line of text. Missing system fonts change glyphs too (fonts and emoji in Puppeteer).
• Lazy images and third parties. Images below the fold, ads, chat widgets and cookie banners. Load or block them deliberately (full-page screenshots shows lazy loading).
• Viewport and pixel ratio. Fix both in code; a different deviceScaleFactor changes every pixel.
• The environment. Generate baselines and run comparisons in the same Docker image, in CI. Our two-image test shows what happens otherwise: hundreds of pixels differ in plain text with nothing changed. A baseline made on a laptop is likely to fail in CI.

Hiding an element with display: none changes the layout below it; covering it with a solid box of the same size keeps the layout. For a timestamp in a fixed-size box, either works; for a banner of varying height, cover it.

Choosing a threshold

The pixelmatch README describes threshold as "a per-pixel perceptual color difference" from 0 to 1, with 0.1 as the default — "not a share of the image". In our test the button colour changing from #3366cc to #2f5fc4 (a small change, the kind a design token edit makes) counted as 0 differing pixels at 0.1 and at 0.2. A plain 200×100 block of those two colours differed fully at threshold 0.01 and not at all at 0.05.

So two separate decisions:

1. How different must a pixel be? In a fixed environment, 0 or close to it — the environment is the same, so any colour difference is real.
2. How many pixels may differ? Allow a small number for anti-aliasing (pixelmatch already ignores pixels it detects as anti-aliased unless includeAA is set), and fail above it.

If your environments differ (developers' machines, several CI runners), raising the threshold hides the noise and real changes with it. Unify the environment instead.

Playwright's toHaveScreenshot

If you test with Playwright, its toHaveScreenshot does most of the above for you: it writes the baseline on the first run, stores one per browser and platform, retries until two consecutive screenshots match, and by default disables CSS animations and transitions (animations: "disabled") and hides the caret (caret: "hide"). The mask option covers elements with a solid box, and stylePath injects your own CSS before the shot. Its threshold defaults to 0.2 — in our test that also missed the button colour change — so set threshold, maxDiffPixels or maxDiffPixelRatio deliberately. The Playwright docs warn, as our measurement shows, that rendering varies with the OS, settings and hardware, and recommend running in the environment where the baselines were made.

With Puppeteer you assemble the same pieces yourself, as in the script above. That is a few dozen lines, and it suits teams who already use Puppeteer for other work. Python teams can do the same with Playwright for Python (Python screenshots).

Where a screenshot API fits

Visual regression tests mostly run against pages that are not public yet — a branch preview, localhost, a component in Storybook — and those belong to your own Puppeteer or Playwright in CI. Our screenshot API takes screenshots of public pages only, so it fits a different job: watching production. A daily check of your public pages, from one consistent browser in one place, catches the change that slipped through — a broken stylesheet after a CDN change, a cookie banner covering the page, a third-party widget pushing content down.

Things to know if you use it that way:

• The same URL with the same options within 24 hours returns the existing file, which suits a daily check. To force a new shot, change the URL, for example with a ?check=2026-11-07 query parameter your site ignores.
• hide_selectors hides changing parts (display: none, so the layout below moves up), wait_for waits for an element, block_cookie_banners and block_ads remove noise, and reduced_motion emulates prefers-reduced-motion.
• A viewport shot costs 1 render, a full-page one 2 (pricing).

The comparison itself — pixelmatch or any image diff — stays in your script: we take the screenshots, we do not keep baselines or report differences.

Checklist

1. Same browser, fonts and image for the baseline and every run; ideally one Docker image in CI.
2. Fixed viewport and device pixel ratio.
3. Animations, transitions and the caret off; fonts loaded; lazy content loaded or blocked.
4. Changing data frozen, replaced or covered.
5. Image sizes compared before pixels.
6. A low colour threshold and an explicit number of allowed pixels.
7. A diff image saved with every failure.