Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
6c906c8
randomize region used for evals, split out pnpm and turbo cache, veri…
pirate Jan 26, 2026
4ccf598
use github actions native chromium for integration tests
pirate Jan 26, 2026
20b6745
restore playwright browser usage for integration tests
pirate Jan 26, 2026
e07bd93
add code coverage and flaky test reporting
pirate Jan 26, 2026
e4e2d63
fix lint
pirate Jan 26, 2026
ecc132d
make sure ctrf artifacts are unique
pirate Jan 26, 2026
2992f77
add missing packages and fix ctrf summaries
pirate Jan 26, 2026
df3708c
fix sanitization of vitest artifacts
pirate Jan 26, 2026
68d79c2
remove custom action for vitest sanitization
pirate Jan 26, 2026
5001deb
only merge coverage in final step
pirate Jan 26, 2026
543bf42
check ratelimits
pirate Jan 26, 2026
1faba4f
log if ratelimits
pirate Jan 26, 2026
e632cb6
limit github api calls
pirate Jan 26, 2026
982699f
fix e2e bb tests running on local and coverage reporting
pirate Jan 27, 2026
df27619
fix chromium path used by integration tests
pirate Jan 27, 2026
fd2385a
make sure we disable sandbox in e2e local tests
pirate Jan 27, 2026
4a1f543
tune coverage reports
pirate Jan 27, 2026
524f87e
fix chromium version used
pirate Jan 27, 2026
8ef22d1
bump test timeouts to improve flakyness when testing against remote
pirate Jan 27, 2026
538081b
fix integration test failures
pirate Jan 27, 2026
a2c8d78
fix flushing of server test output
pirate Jan 27, 2026
3002cb7
allow playwright downloads in server integration tests
pirate Jan 27, 2026
80e6e8a
increase screenshot timeout on prod bb browser
pirate Jan 27, 2026
0efeb40
force disableAPI true in all e2e tests
pirate Jan 27, 2026
da42c9b
rename TEST_ENV to STAGEHAND_ENV
pirate Jan 27, 2026
e5a62fc
up timetouts in screenshot test to 5s
pirate Jan 27, 2026
ad422cb
improve env reporter in tests
pirate Jan 27, 2026
a354cee
lint
pirate Jan 27, 2026
d60f86a
fix start integration tests not using CHROME_PATH
pirate Jan 27, 2026
bc24428
use non-pretty logs for SEA in CI
pirate Jan 27, 2026
1fbacd6
lint
pirate Jan 27, 2026
ec4af40
fix init script tests being broken by tsx-loader changing
pirate Jan 27, 2026
742e76b
force stagehand-server to use github actions chromium
pirate Jan 27, 2026
8854ee3
fix init script and click test flakyness
pirate Jan 27, 2026
821d4d9
tweak screenshot timeout in CI
pirate Jan 27, 2026
7bfce55
fix cubic comment about testing init script args
pirate Jan 27, 2026
c3c0069
fix eslint
pirate Jan 27, 2026
ead4590
enable sourcemaps and coverage for SEA binaries
pirate Jan 27, 2026
6513e88
fix eval task result logging
pirate Jan 27, 2026
80fad7e
add c8 deve dependency
pirate Jan 27, 2026
3819ec6
fix rename
pirate Jan 27, 2026
2b557d8
Merge branch 'main' into parallel-ci-tests
pirate Feb 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions .github/actions/publish-ctrf-report/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
name: Publish CTRF report
description: Convert JUnit to CTRF (optional) and publish CTRF report with Insights.
inputs:
junit-path:
description: Path to the JUnit XML report.
required: false
default: ""
tool:
description: Tool name for the CTRF report (e.g. playwright, vitest, node-test).
required: false
default: "junit"
github-token:
description: GitHub token for publishing CTRF reports.
required: true

outputs:
artifact-name:
description: Name of the CTRF artifact uploaded by the reporter.
value: ${{ steps.ctrf-meta.outputs.ARTIFACT_NAME }}
report-path:
description: CTRF report path or glob.
value: ${{ steps.ctrf-meta.outputs.REPORT_PATH }}

runs:
using: composite
steps:
- name: Set CTRF metadata
id: ctrf-meta
shell: bash
run: |
junit_path="${{ inputs.junit-path }}"
if [ -n "$junit_path" ]; then
rel_path="${junit_path##*/ctrf/}"
rel_path="${rel_path%.xml}"
safe_name="${rel_path//\//-}"
echo "ARTIFACT_NAME=ctrf-${safe_name}" >> "$GITHUB_OUTPUT"
echo "REPORT_PATH=ctrf/${safe_name}.json" >> "$GITHUB_OUTPUT"
else
job_name="${GITHUB_JOB:-ctrf-report}"
safe_job_name="${job_name//\//-}"
echo "ARTIFACT_NAME=ctrf-${safe_job_name}" >> "$GITHUB_OUTPUT"
echo "REPORT_PATH=./ctrf/*.json" >> "$GITHUB_OUTPUT"
fi

- name: Convert JUnit to CTRF
if: ${{ inputs.junit-path != '' }}
shell: bash
run: |
junit_path="${{ inputs.junit-path }}"
if [ -f "$junit_path" ]; then
if [ ! -s "$junit_path" ]; then
echo "JUnit report is empty at $junit_path"
exit 0
fi
if ! pnpm dlx junit-to-ctrf "$junit_path" \
-o "${{ steps.ctrf-meta.outputs.REPORT_PATH }}" \
-t "${{ inputs.tool }}"; then
echo "JUnit-to-CTRF conversion failed; leaving no CTRF report."
exit 0
fi
else
echo "JUnit report not found at $junit_path"
exit 0
fi

- name: Check CTRF report
id: ctrf-report
shell: bash
run: |
report_path="${{ steps.ctrf-meta.outputs.REPORT_PATH }}"
if [ -f "$report_path" ]; then
echo "found=true" >> "$GITHUB_OUTPUT"
else
echo "found=false" >> "$GITHUB_OUTPUT"
fi

- name: Upload CTRF report artifact
if: ${{ steps.ctrf-report.outputs.found == 'true' }}
uses: actions/upload-artifact@v4
with:
name: ${{ steps.ctrf-meta.outputs.ARTIFACT_NAME }}
path: ${{ steps.ctrf-meta.outputs.REPORT_PATH }}
61 changes: 61 additions & 0 deletions .github/actions/select-browserbase-region/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Select Browserbase region
description: Select a Browserbase region based on a weighted distribution.
inputs:
distribution:
description: Comma-separated region=weight list (e.g. us-west-2=40,us-east-1=20).
required: true
outputs:
region:
description: Selected region.
value: ${{ steps.select.outputs.region }}
runs:
using: composite
steps:
- id: select
shell: bash
run: |
dist="${{ inputs.distribution }}"
if [ -z "$dist" ]; then
echo "BROWSERBASE_REGION_DISTRIBUTION is empty"
exit 1
fi
IFS=',' read -r -a entries <<< "$dist"
total=0
regions=()
weights=()
for entry in "${entries[@]}"; do
region="${entry%%=*}"
weight="${entry#*=}"
if [ -z "$region" ] || [ -z "$weight" ]; then
echo "Invalid region distribution entry: $entry"
exit 1
fi
if ! [[ "$weight" =~ ^[0-9]+$ ]]; then
echo "Invalid weight for region $region: $weight"
exit 1
fi
regions+=("$region")
weights+=("$weight")
total=$((total + weight))
done
if [ "$total" -le 0 ]; then
echo "Invalid total weight: $total"
exit 1
fi
roll=$((RANDOM % total))
cumulative=0
chosen=""
for i in "${!regions[@]}"; do
cumulative=$((cumulative + weights[i]))
if [ "$roll" -lt "$cumulative" ]; then
chosen="${regions[i]}"
break
fi
done
if [ -z "$chosen" ]; then
echo "Failed to choose Browserbase region"
exit 1
fi
echo "Selected Browserbase region: $chosen"
echo "region=$chosen" >> "$GITHUB_OUTPUT"
echo "BROWSERBASE_REGION=$chosen" >> "$GITHUB_ENV"
51 changes: 51 additions & 0 deletions .github/actions/setup-node-pnpm-turbo/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Setup Node, pnpm, and Turbo cache
description: Configure pnpm and Node.js with caching, restore Turbo cache, and install dependencies.
inputs:
node-version:
description: Node.js version to use.
required: false
default: "20.x"
use-prebuilt-artifacts:
description: Whether to download pre-built package from build artifacts.
required: false
default: "true"

runs:
using: composite
steps:
- uses: pnpm/action-setup@v4

- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: ${{ inputs.node-version }}
cache: 'pnpm'
cache-dependency-path: '**/pnpm-lock.yaml'

- name: Restore Turbo cache
uses: actions/cache/restore@v4
with:
path: .turbo
key: ${{ runner.os }}-turbo-${{ hashFiles('turbo.json', '**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-turbo-

- name: Install dependencies
shell: bash
run: pnpm install --frozen-lockfile --prefer-offline

- name: Download build artifacts
if: ${{ inputs.use-prebuilt-artifacts == 'true' }}
uses: actions/download-artifact@v4
with:
name: build-artifacts
path: .
merge-multiple: true

- name: Prepare test output directories
shell: bash
run: |
mkdir -p "${GITHUB_WORKSPACE}/ctrf"
if [ -n "${NODE_V8_COVERAGE:-}" ]; then
mkdir -p "$NODE_V8_COVERAGE"
fi
22 changes: 22 additions & 0 deletions .github/actions/upload-v8-coverage/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: Upload V8 coverage
description: Upload V8 coverage artifacts.
inputs:
name:
description: Artifact name.
required: true

runs:
using: composite
steps:
- name: Normalize artifact name
id: normalize
shell: bash
run: |
name="${{ inputs.name }}"
echo "name=${name//\//-}" >> "$GITHUB_OUTPUT"

- name: Upload coverage artifact
uses: actions/upload-artifact@v4
with:
name: ${{ steps.normalize.outputs.name }}
path: ${{ env.NODE_V8_COVERAGE }}
Loading
Loading