Replace Percy with local odiff-based visual regression testing

[nicksteffens]

Summary

Replace Percy visual regression testing with a local, cost-effective solution using odiff for image comparison. This repo will serve as the proving ground for both Ember test and Storybook migration patterns before rolling out to front-end/canvas and other repos.

Current State

• 56 Percy snapshot calls across 14 test files
• Percy helper at tests/helpers/percy-snapshot.js with auto-naming pattern
• Storybook configured with @storybook/ember
• testem.js already launches Chrome with --remote-debugging-port

Proposed Solution

1. Ember Component Tests → Custom Screenshot Addon

Create an Ember addon that uses Puppeteer via testem middleware to capture real Chrome screenshots:

Architecture:

• Testem middleware runs Puppeteer server-side
• Connects to Chrome's remote debugging port
• Tests call simple await screenshot(assert) API
• Screenshots saved locally to tests/snapshots/

Required Changes:

// testem.js
- '--remote-debugging-port=0',
+ '--remote-debugging-port=9222',

Migration (codemod-able):

// Before
import percySnapshot from '@percy/ember';
await percySnapshot(assert);
await percySnapshot(assert, 'with label');

// After
import { screenshot } from 'ember-screenshot';
await screenshot(assert);
await screenshot(assert, 'with label');

2. Storybook → Storycap + odiff

Replace @percy/storybook with Storycap:

# Capture screenshots
npx storycap http://localhost:6006 -o ./storybook-snapshots

# Compare with odiff in CI
npx odiff ./storybook-snapshots/baseline ./storybook-snapshots/current ./storybook-snapshots/diff

3. CI Integration → GitHub Action

Use a custom GitHub Action (to be published) that:

• Compares snapshots between PR branch and baseline branch
• Uses odiff for pixel-level comparison with configurable threshold
• Generates visual diff report in GitHub Actions job summary
• Uploads diff images as artifacts

Tasks

• Update testem.js to use fixed remote debugging port (9222)
• Create/integrate ember-screenshot addon with Puppeteer middleware
• Add screenshot helper with same naming pattern as Percy wrapper
• Migrate Percy calls to new screenshot helper (56 calls)
• Set up Storycap for Storybook screenshot capture
• Integrate odiff GitHub Action for CI comparison
• Remove @percy/ember and @percy/cli dependencies
• Update CI workflow to use new visual testing approach
• Document migration process for other repos

Benefits

• Cost savings: No per-snapshot pricing
• Local snapshots: Stored in repo, version controlled
• Faster feedback: No waiting for Percy cloud processing
• Test runner agnostic: Same odiff action works across Cypress, Storybook, Ember

References

• odiff - SIMD-optimized image comparison
• Storycap - Storybook screenshot capture
• Percy replacement GitHub Project: https://github.com/users/nicksteffens/projects/1