Skip to content

Fix/test playwright #223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mishautkin wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Fix/test playwright #223

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
eab074b
Add WORK_DIR input
mishautkin Feb 13, 2026
93915dc
Add GH_TOKEN to PRE_SCRIPT
mishautkin Feb 17, 2026
9ae5c30
Add PLAYWRIGHT_GREP and PLAYWRIGHT_PROJECT inputs
mishautkin Feb 17, 2026
518e6ff
Update documentation
mishautkin Feb 17, 2026
51b8515
Move .env.ci creation to a separate step
mishautkin Feb 17, 2026
dd6b433
Add PLAYWRIGHT_GREP_INVERT input for custom commands
mishautkin Feb 17, 2026
2b101e0
Update documentation for `--grep-invert`
mishautkin Feb 17, 2026
4ecd866
Fix quotes in playwright custom commands
mishautkin Feb 17, 2026
d7aad04
Lint md fix
mishautkin Feb 17, 2026
9376943
Move GH_TOKEN to PR_SCRIPT step
mishautkin Feb 19, 2026
028f13b
Replace PLAYWRIGHT_ inputs with a single PLAYWRIGHT_CLI_ARGS
mishautkin Feb 19, 2026
9b99dc9
Update documentation for PLAYWRIGHT_CLI_ARGS
mishautkin Feb 25, 2026
14c2607
Lint md fix
mishautkin Feb 25, 2026
6a6d159
Update test-playwright.yml
mishautkin Mar 5, 2026
5e5e67a
Remove PLAYWRIGHT_CLI_ARGS input
mishautkin Mar 5, 2026
6637d73
Merge branch 'fix/test-playwright' of https://github.com/inpsyde/reus…
mishautkin Mar 5, 2026
3b962f5
Rename SCRIPT_NAME to PLAYWRIGHT_SCRIPT
mishautkin Mar 5, 2026
f6a50e1
Increase job timeout
mishautkin Mar 10, 2026
e293364
Update default node version
mishautkin Mar 12, 2026
334d56a
Update docs
mishautkin Mar 12, 2026
58844e3
Add COMPOSER_AUTH to Playwright workflow PRE_SCRIPT environment variable
Biont Mar 18, 2026
0481294
Add wp-env boot step to Playwright workflow
Biont Mar 26, 2026
a73d9c2
Reorder wp-env boot step in Playwright workflow for better execution …
Biont Apr 1, 2026
8daf275
Add environment variable loading step to wp-env boot in Playwright wo…
Biont Apr 1, 2026
53acfeb
Add Ngrok setup
mishautkin Apr 16, 2026
2796404
Update docs with Ngrok
mishautkin Apr 16, 2026
a0b0a95
Rename NGROK_AUTHTOKEN to NGROK_AUTH_TOKEN
mishautkin Apr 17, 2026
74cbc77
Merge pull request #239 from inpsyde/fix/test-playwright-ngrok
mishautkin Apr 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 93 additions & 9 deletions .github/workflows/test-playwright.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ on:
required: false
NODE_VERSION:
description: Node version with which the node script will be executed.
default: 18
default: '24'
required: false
type: string
NPM_REGISTRY_DOMAIN:
Expand All @@ -57,9 +57,20 @@ on:
default: ''
required: false
type: string
SCRIPT_NAME:
description: The name of a custom script to run the tests.
required: true
PLAYWRIGHT_SCRIPT:
description: The name of a custom npm script to run the tests.
default: ''
required: false
type: string
WORK_DIR:
Copy link
Copy Markdown
Contributor

@dottxado dottxado Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question

During the last meeting, we agreed to move all the npm dependencies to the root package.json to avoid this kind of complication, since there is no evidence that your dependencies will cause version clashes. Can you please explain why you are introducing this input? 😄

Copy link
Copy Markdown
Author

@mishautkin mishautkin Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've explained this in the first point of "What is the new behavior" section. But maybe I can find a workaround, e.g. by switching the tests: dir in playwright.config 🤔

Copy link
Copy Markdown
Member

@Biont Biont Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is going to be hard to pull off without also assuming control of the playwright.config.js. From my (limited) experience, Playwright is pretty stoic about the way it is configured and bootstraps itself from the config file with extremely limited entrypoints for external configuration/overrides.

We could use npm workspaces to isolate Playwright while also satisfying the centralized node_modules/ requirement, but I am not sure if that really reduces the complexity of the setup. It would allow us to install playwright in the root while executing it in WORK_DIR.

But this still requires the project to be structured to suit the workflow (instead of configuring the workflow to suit the project within a reasonable framework)
It might be a middleground here though.

Another idea:
Playwright supports pointing to a config file with --config - is it then also resolving and picking up tests relative to the config file? Then we would only need a PLAYWRIGHT_CONFIG input

Copy link
Copy Markdown
Author

@mishautkin mishautkin Feb 25, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I remember about this unresolved comment. Moving playwright and all it's dependencies from tests/qa/ to root takes some time. I need to go through some PRs on Mollie before I can tackle this.

In case of several PW projects (like on PayPal with legacy UI), my plan is to manage testDir in playwright.config using path stored in env var PLAYWRIGHT_TEST_DIR='tests/qa/tests'.

description: Working directory for npm install, Playwright install, and test execution.
default: '.'
required: false
type: string
NGROK_DOMAIN:
description: Reserved ngrok domain for the tunnel (paid account). Required when NGROK_AUTH_TOKEN is provided.
default: ''
required: false
type: string
secrets:
ENV_FILE_DATA:
Expand All @@ -80,10 +91,13 @@ on:
COMPOSER_AUTH_JSON:
description: Authentication for privately hosted packages and repositories as a JSON formatted object.
required: false
NGROK_AUTH_TOKEN:
description: Ngrok auth token for tunneling. When set, ngrok is installed and a tunnel is started.
required: false

jobs:
run-playwright-test:
timeout-minutes: 60
timeout-minutes: 120
runs-on: ubuntu-latest
env:
PHP_CHECK: false
Expand Down Expand Up @@ -111,7 +125,8 @@ jobs:

- name: Set up node cache mode
run: |
if [ -f "${GITHUB_WORKSPACE}/package-lock.json" ] || [ -f "${GITHUB_WORKSPACE}/npm-shrinkwrap.json" ]; then
WORK_DIR="${{ inputs.WORK_DIR }}"
if [ -f "${GITHUB_WORKSPACE}/${WORK_DIR}/package-lock.json" ] || [ -f "${GITHUB_WORKSPACE}/${WORK_DIR}/npm-shrinkwrap.json" ]; then
echo "NODE_CACHE_MODE=npm" >> $GITHUB_ENV
else
echo "No lock files found or unknown package manager"
Expand Down Expand Up @@ -143,26 +158,95 @@ jobs:
cache: ${{ env.NODE_CACHE_MODE }}

- name: Install dependencies
working-directory: ${{ inputs.WORK_DIR }}
run: npm ${{ env.NODE_CACHE_MODE == 'npm' && 'ci' || 'install' }}

- name: Install Playwright dependencies
working-directory: ${{ inputs.WORK_DIR }}
run: |
npx playwright install ${{ inputs.PLAYWRIGHT_BROWSER_ARGS }}

- name: Create environment file
working-directory: ${{ inputs.WORK_DIR }}
env:
ENV_FILE_DATA: ${{ secrets.ENV_FILE_DATA }}
run: printenv ENV_FILE_DATA > .env.ci

- name: Boot wp-env
if: ${{ hashFiles('.wp-env.json') != '' }}
run: |
set -a && source .env.ci && set +a
npm -g i @wordpress/env && wp-env start

- name: Install ngrok
env:
NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }}
if: ${{ env.NGROK_AUTH_TOKEN != '' }}
run: |
curl -sSL https://ngrok-agent.s3.amazonaws.com/ngrok.asc \
| sudo tee /etc/apt/trusted.gpg.d/ngrok.asc >/dev/null
echo "deb https://ngrok-agent.s3.amazonaws.com buster main" \
| sudo tee /etc/apt/sources.list.d/ngrok.list
sudo apt-get update && sudo apt-get install -y ngrok

- name: Start ngrok tunnel
env:
NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }}
NGROK_DOMAIN: ${{ inputs.NGROK_DOMAIN }}
if: ${{ env.NGROK_AUTH_TOKEN != '' }}
working-directory: ${{ inputs.WORK_DIR }}
run: |
ngrok config add-authtoken "$NGROK_AUTH_TOKEN"

DOMAIN_FLAG=""
if [ -n "$NGROK_DOMAIN" ]; then
DOMAIN_FLAG="--domain=$NGROK_DOMAIN"
fi

ngrok http 80 $DOMAIN_FLAG --log=stdout --log-level=info > /tmp/ngrok.log 2>&1 &
sleep 3

NGROK_URL=$(curl -s http://127.0.0.1:4040/api/tunnels \
| jq -r '.tunnels[] | select(.proto=="https") | .public_url')
echo "Ngrok URL: $NGROK_URL"

if [ -z "$NGROK_URL" ]; then
echo "::error::Failed to get ngrok tunnel URL"
cat /tmp/ngrok.log
exit 1
fi

echo "NGROK_URL=$NGROK_URL" >> "$GITHUB_ENV"

- name: Update WordPress URLs to ngrok
env:
NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }}
if: ${{ env.NGROK_AUTH_TOKEN != '' }}
working-directory: ${{ inputs.WORK_DIR }}
run: |
wp-env run tests-cli wp config set WP_SITEURL "$NGROK_URL"
wp-env run tests-cli wp config set WP_HOME "$NGROK_URL"
sed -i "s|WP_BASE_URL=.*|WP_BASE_URL=$NGROK_URL|" .env.ci

- name: Execute custom code before executing the test script
working-directory: ${{ inputs.WORK_DIR }}
env:
GH_TOKEN: ${{ github.token }}
COMPOSER_AUTH: '${{ secrets.COMPOSER_AUTH_JSON }}'
run: |
set -a && source .env.ci && set +a
Comment thread
mishautkin marked this conversation as resolved.
${{ inputs.PRE_SCRIPT }}


- name: Run script for test
id: run-script
continue-on-error: true
working-directory: ${{ inputs.WORK_DIR }}
run: |
touch .env.ci
echo "${{ secrets.ENV_FILE_DATA }}" >> .env.ci
# Ensure .env.ci is deleted on exit
trap 'rm -f .env.ci' EXIT

npm run ${{ inputs.SCRIPT_NAME }}
npm run ${{ inputs.PLAYWRIGHT_SCRIPT }}

- name: Upload artifact
if: always()
Expand Down
178 changes: 163 additions & 15 deletions docs/test-playwright.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ This workflow executes Playwright-based tests in a controlled and isolated envir
The workflow can:

- execute a building step, both for node and PHP environments (if the PHP version is provided and a `composer.json` file is present)
- create an environment variables file named `.env.ci` dedicated to the test step; load this file using `dotenv-ci` directly in your test script, e.g., `./node_modules/.bin/dotenv -e .env.ci -- npm run e2e`
- execute the tests using Playwright
- upload the artifacts
- create an environment variables file named `.env.ci` dedicated to the test step; load this file using `dotenv-ci` directly in your test script, e.g., `./node_modules/.bin/dotenv -e .env.ci -- npm run e2e`. The file is also sourced before `PRE_SCRIPT`, making all variables available as environment variables.
- execute the tests using Playwright via a custom npm script.
- upload the artifacts.
- optionally start an [ngrok](https://ngrok.com/) tunnel for webhook delivery to `wp-env` environments (when `NGROK_AUTH_TOKEN` is provided).

**Simplest possible example:**

Expand All @@ -21,7 +22,7 @@ jobs:
uses: inpsyde/reusable-workflows/.github/workflows/test-playwright.yml@main
with:
ARTIFACT_PATH: './artifacts'
SCRIPT_NAME: 'ci-test-e2e'
PLAYWRIGHT_SCRIPT: 'ci-test-e2e'
```

## Configuration parameters
Expand All @@ -36,25 +37,41 @@ jobs:
| `ARTIFACT_PATH` | | A file, directory or wildcard pattern that describes what to upload |
| `ARTIFACT_RETENTION_DAYS` | `30` | Duration after which artifact will expire in day |
| `COMPOSER_DEPS_INSTALL` | `false` | Whether to install Composer dependencies |
| `NODE_VERSION` | `18` | Node version with which the node script will be executed |
| `NGROK_DOMAIN` | `''` | Reserved ngrok domain for the tunnel (paid account). Required when `NGROK_AUTH_TOKEN` is provided |
| `NODE_VERSION` | `24` | Node version with which the node script will be executed |
| `NPM_REGISTRY_DOMAIN` | `'https://npm.pkg.github.com/'` | Domain of the private npm registry |
| `PHP_VERSION` | `'8.2'` | PHP version with which the dependencies are installed |
| `PLAYWRIGHT_BROWSER_ARGS` | `'--with-deps'` | Set of arguments passed to `npx playwright install` |
| `PRE_SCRIPT` | `''` | Run custom shell code before executing the test script |
| `SCRIPT_NAME` | | The name of a custom script to run the tests |
| `PRE_SCRIPT` | `''` | Run custom shell code before executing the test script. `GH_TOKEN` and all `ENV_FILE_DATA` variables are available |
| `PLAYWRIGHT_SCRIPT` | `''` | The name of a custom npm script to run the tests. |
| `WORK_DIR` | `'.'` | Working directory for npm install, Playwright install, PRE_SCRIPT, and test execution |


### Secrets

| Name | Description |
|-----------------------|------------------------------------------------------------------------------------------|
| `ENV_FILE_DATA` | Additional environment variables for the tests |
| `COMPOSER_AUTH_JSON` | Authentication for privately hosted packages and repositories as a JSON formatted object |
| `NPM_REGISTRY_TOKEN` | Authentication for the private npm registry |
| `ENV_FILE_DATA` | Additional environment variables for the tests. Also sourced before `PRE_SCRIPT` |
| `GITHUB_USER_EMAIL` | Email address for the GitHub user configuration |
| `GITHUB_USER_NAME` | Username for the GitHub user configuration |
| `GITHUB_USER_SSH_KEY` | Private SSH key associated with the GitHub user passed as `GITHUB_USER_NAME` |
| `NGROK_AUTH_TOKEN` | Ngrok auth token. When set, ngrok is installed and a tunnel is started before tests run |
| `NPM_REGISTRY_TOKEN` | Authentication for the private npm registry |

## Ngrok tunnel

When `NGROK_AUTH_TOKEN` is provided, the workflow automatically:

1. Installs ngrok on the runner.
2. Starts an HTTPS tunnel to port 80 using the reserved domain from `NGROK_DOMAIN`.
3. Updates `WP_SITEURL`, `WP_HOME` (via `wp-env`) and `WP_BASE_URL` (in `.env.ci`) to the tunnel URL.

This runs **after** `wp-env` boots and **before** `PRE_SCRIPT`, so webhooks from external services (e.g. payment gateways) can reach the test environment.

Requires a [paid ngrok account](https://ngrok.com/pricing) with a reserved domain.

**Example with configuration parameters:**
## Example with configuration parameters

```yml
name: E2E Testing
Expand All @@ -72,7 +89,7 @@ jobs:
artifacts/*
playwright-report/
ARTIFACT_INCLUDE_HIDDEN_FILES: true
SCRIPT_NAME: 'ci-test-e2e'
PLAYWRIGHT_SCRIPT: 'ci-test-e2e'
COMPOSER_DEPS_INSTALL: true
PHP_VERSION: ${{ matrix.php }}
NODE_VERSION: 20
Expand All @@ -88,11 +105,142 @@ jobs:
NPM_REGISTRY_TOKEN: ${{ secrets.DEPLOYBOT_PACKAGES_READ_ACCESS_TOKEN}}
```

**Example of secrets:**
## Example with ngrok tunnel

```yml
name: E2E Testing

on:
workflow_dispatch:
jobs:
e2e-playwright:
uses: inpsyde/reusable-workflows/.github/workflows/test-playwright.yml@main
with:
ARTIFACT_PATH: |
artifacts/*
playwright-report/
PLAYWRIGHT_SCRIPT: 'ci-test-e2e'
NODE_VERSION: 24
PLAYWRIGHT_BROWSER_ARGS: 'chromium --with-deps'
NGROK_DOMAIN: ${{ secrets.NGROK_DOMAIN }}
PRE_SCRIPT: |
gh run download ${{ github.run_id }} -p "my-plugin-*" -D resources/files
npm run setup:env
secrets:
ENV_FILE_DATA: ${{ secrets.ENV_FILE_DATA }}
NPM_REGISTRY_TOKEN: ${{ secrets.DEPLOYBOT_PACKAGES_READ_ACCESS_TOKEN }}
NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }}
```

## Example with custom inputs

```yml
name: E2E Testing

on:
workflow_dispatch:
inputs:
TEST_SUITE:
description: 'Test suite to run'
required: true
default: 'critical'
type: choice
options:
- smoke
- critical
- all

jobs:
e2e-playwright:
uses: inpsyde/reusable-workflows/.github/workflows/test-playwright.yml@main
with:
WORK_DIR: 'tests/qa'
ARTIFACT_PATH: |
tests/qa/artifacts/*
tests/qa/playwright-report/
PLAYWRIGHT_SCRIPT: ${{ inputs.TEST_SUITE }}
NODE_VERSION: 22
PLAYWRIGHT_BROWSER_ARGS: 'chromium --with-deps'
PRE_SCRIPT: |
gh run download ${{ github.run_id }} -p "my-plugin-*" -D resources/files
npm run setup:env
secrets:
ENV_FILE_DATA: ${{ secrets.ENV_FILE_DATA }}
NPM_REGISTRY_TOKEN: ${{ secrets.DEPLOYBOT_PACKAGES_READ_ACCESS_TOKEN}}
```

## Example of secrets

For `ENV_FILE_DATA`:

```SHELL
TEST_EXEC_KEY=YOUR-KEY
WP_BASE_URL=https://example.com
```bash
# playwright-utils config
WP_BASE_URL='http://mywp.site'
WP_USERNAME=admin
WP_PASSWORD=password
WP_BASIC_AUTH_USER=admin
WP_BASIC_AUTH_PASS=password
STORAGE_STATE_PATH='./storage-states'
STORAGE_STATE_PATH_ADMIN='./storage-states/admin.json'
WORDPRESS_DB_USER=root
WORDPRESS_DB_PASSWORD=password

WPCLI_ENV_TYPE= # localhost, vip, wpenv, ddev, ssh
WPCLI_PATH= # for localhost, wpenv
# For WPCLI_ENV_TYPE=ssh
SSH_LOGIN=
SSH_HOST=
SSH_PORT=
SSH_PATH=
# For WPCLI_ENV_TYPE=vip
VIP_APP=
VIP_ENV=

# WooCommerce specific env vars
WC_API_KEY=
WC_API_SECRET=
WC_DEFAULT_COUNTRY=usa
WC_DEFAULT_CURRENCY=USD

# Xray in Jira
XRAY_CLIENT_ID=
XRAY_CLIENT_SECRET=
TEST_EXEC_KEY=

# Ngrok (optional, enables tunnel for webhook delivery)
NGROK_DOMAIN=
NGROK_AUTH_TOKEN=
```

## Examples of `PRE_SCRIPT`

### VIP connection

```bash
PRE_SCRIPT: |
npm install -g @automattic/vip
mkdir -p ~/.config/configstore
echo '{"vip-go-cli": "'"$VIP_TOKEN"'"}' > ~/.config/configstore/vip-go-cli.json
```

### Distributing vars per env

In case of several test environments (staging, production, etc.) `PRE_SCRIPT` can be used to setup env-specific vars from `ENV_FILE_DATA`. For example `PERCY_TOKEN`:

```bash
PRE_SCRIPT: |
echo "PERCY_TOKEN=$PERCY_TOKEN_STAGE" >> "$GITHUB_ENV"
```

### Download and zip plugin artifact

For cases when plugin is built within the same workflow:

```bash
PRE_SCRIPT: |
gh run download ${{ github.run_id }} -p "woocommerce-paypal-payments-*" -D tests/qa/resources/files
cd tests/qa/resources/files
mv woocommerce-paypal-payments-*/woocommerce-paypal-payments .
zip -r woocommerce-paypal-payments.zip woocommerce-paypal-payments
rm -rf woocommerce-paypal-payments woocommerce-paypal-payments-*/
```
2 changes: 1 addition & 1 deletion package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading