From 04ccaa4648191660ce5211d84f5e9f646c97c31e Mon Sep 17 00:00:00 2001 From: marktnoonan Date: Thu, 4 Dec 2025 18:00:51 -0500 Subject: [PATCH 1/3] initial pass at docs for config profiles --- docs/accessibility/configuration/overview.mdx | 52 +++- docs/accessibility/configuration/profiles.mdx | 12 + docs/partials/_attributefilters.mdx | 4 +- docs/partials/_elementfilters.mdx | 7 +- docs/partials/_profiles.mdx | 249 ++++++++++++++++++ docs/partials/_viewfilters.mdx | 4 +- docs/partials/_views.mdx | 4 +- .../allowedinteractioncommands.mdx | 4 +- .../configuration/elementgroups.mdx | 4 +- docs/ui-coverage/configuration/elements.mdx | 4 +- docs/ui-coverage/configuration/overview.mdx | 46 +++- docs/ui-coverage/configuration/profiles.mdx | 12 + src/theme/MDXComponents.js | 2 + 13 files changed, 382 insertions(+), 22 deletions(-) create mode 100644 docs/accessibility/configuration/profiles.mdx create mode 100644 docs/partials/_profiles.mdx create mode 100644 docs/ui-coverage/configuration/profiles.mdx diff --git a/docs/accessibility/configuration/overview.mdx b/docs/accessibility/configuration/overview.mdx index eac51fe0bf..213cead99b 100644 --- a/docs/accessibility/configuration/overview.mdx +++ b/docs/accessibility/configuration/overview.mdx @@ -22,45 +22,79 @@ To add or modify the configuration for your project, navigate to the "App Qualit alt="The Cypress Cloud UI showing the configuration editor" /> -You can use the provided editor to write configuration in JSON format. A complete configuration with all available options looks as follows: +You can use the provided editor to write configuration in JSON format. -```typescript +### Comments + +All configuration objects support an optional `comment` property that you can use to provide context and explanations for why certain values are set. This helps make your configuration easier to understand and maintain, especially when working in teams or revisiting configuration after some time. + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false, + "comment": "Exclude temporary test elements from accessibility reports" + } + ] +} +``` + +### Profiles + +The `profiles` property allows you to use different configuration settings for different runs based on [run tags](/app/references/command-line#cypress-run-tag-lt-tag-gt). See the [Profiles](/accessibility/configuration/profiles) guide for more details. + +### Complete configuration example + +A complete configuration with all available options looks as follows: + +```json { "views": [ { "pattern": string, "groupBy": [ string - ] + ], + "comment": string } ], "viewFilters": [ { "pattern": string, - "include": boolean + "include": boolean, + "comment": string } ], "elementFilters": [ { "selector": string, - "include": boolean + "include": boolean, + "comment": string } ], "significantAttributes": [ string - ] + ], "attributeFilters": [ { "attribute": string, "value": string, - "include": boolean + "include": boolean, + "comment": string + } + ], + "profiles": [ + { + "name": string, + "config": { + // Any App Quality configuration options + } } ] } ``` -Note that these root-level App Quality configuration properties (`elementFilters`, `views`, and `viewFilters`) impact both UI Coverage and Accessibility. - ### Viewing Configuration for a Run You can view configuration information for each run in the Properties tab, as shown below. This is the configuration set for the project at the start of the run. diff --git a/docs/accessibility/configuration/profiles.mdx b/docs/accessibility/configuration/profiles.mdx new file mode 100644 index 0000000000..43cf2bf252 --- /dev/null +++ b/docs/accessibility/configuration/profiles.mdx @@ -0,0 +1,12 @@ +--- +sidebar_label: profiles +title: 'Profiles | Cypress Accessibility' +description: 'The `profiles` configuration property allows you to create configuration overrides that are applied based on run tags.' +sidebar_position: 100 +--- + + + +# profiles + + diff --git a/docs/partials/_attributefilters.mdx b/docs/partials/_attributefilters.mdx index 91474d0613..acff3db8cf 100644 --- a/docs/partials/_attributefilters.mdx +++ b/docs/partials/_attributefilters.mdx @@ -24,7 +24,8 @@ supported, if you need to split them up. { "attribute": string, "value": string, - "include": boolean + "include": boolean, + "comment": string } ] } @@ -39,6 +40,7 @@ For every attribute that an element has, the first `attributeFilters` rule for w | `attribute` | Required | | A regex string to match attribute names | | `value` | Optional | `.*` | A regex string to match attribute values | | `include` | Optional | `true` | A boolean to specify whether the matched attribute should be included. | +| `comment` | Optional | | A comment describing the purpose of this filter rule. | ## Examples diff --git a/docs/partials/_elementfilters.mdx b/docs/partials/_elementfilters.mdx index a71b966a0a..ab15d81d35 100644 --- a/docs/partials/_elementfilters.mdx +++ b/docs/partials/_elementfilters.mdx @@ -18,7 +18,8 @@ supported, if you need to split them up. "elementFilters": [ { "selector": string, - "include": boolean + "include": boolean, + "comment": string } ] } @@ -32,6 +33,7 @@ The first `elementFilters` rule for which the selector property matches the elem | ---------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------- | | `selector` | Required | | A CSS selector to identify elements. Supports standard CSS selector syntax, including IDs, classes, attributes, and combinators. | | `include` | Optional | `true` | A boolean indicating whether the matched elements should be included in the report. | +| `comment` | Optional | | A comment describing the purpose of this filter rule. | ## Examples @@ -44,7 +46,8 @@ The first `elementFilters` rule for which the selector property matches the elem "elementFilters": [ { "selector": "#button-2", - "include": false + "include": false, + "comment": "Exclude test-only button from reports" } ] } diff --git a/docs/partials/_profiles.mdx b/docs/partials/_profiles.mdx new file mode 100644 index 0000000000..50e379dab6 --- /dev/null +++ b/docs/partials/_profiles.mdx @@ -0,0 +1,249 @@ +The `profiles` property allows you to create configuration overrides that are applied based on run tags. This enables you to use different configuration settings for different types of runs, such as regression tests compared to smoke tests environments, or providing scoped results relevant to specific teams. + +## Why use profiles? + +- **Team specific reporting**: If multiple teams use the same Cypress Cloud project, they may have completely different areas of concern for which pages or DOM elements are included in reports about accessibility or UI Coverage. Profiles allow each team to see and track against only the results that matter to them, and remove all other findings. +- **Run-type customization**: Use different filters or settings for regression runs versus pull requests. It can be useful to have a narrow config for blocking a merge, optimized for the most critical areas of your app -- while still running a wide config on a full regression suite to manage on a difference cadence. +- **Skip runs when needed**: If you know that certain kinds of runs are not going to be valuable for App Quality reporting, you can ignore all view on these runs so that no report is created. In some situations this can improve clarity about when to look at a report and which reports are considered significant. + +## How profiles work + +Profiles are selected by matching run tags to profile names. When you run Cypress with the [`--tag`](/app/references/command-line#cypress-run-tag-lt-tag-gt) flag, Cypress Cloud looks for a profile whose `name` matches one of the tags. If a match is found, properties defined in profile's `config` properties override the root configuration. Properties defined in the root that are not referenced in the profile will be inherited, meaning you do not need to repeat config values that you want to keep the same. + +If more than one tag provided to a run matches a profile in your App Quality profiles array, the first matching profile in the array will be used. The order in which the tags are passed to the run doesn't matter. + +## Best practices + +Use a naming convention like `aq-config-x` (e.g., `aq-config-regression`, `aq-config-staging`) to make it clear that a tag is used for configuration lookup purposes. + +While relying on existing tags works just fine, but being explicit will help avoid accidental removal or changes of important tags. + +## Syntax + +```json +{ + "profiles": [ + { + "name": string, + "config": { + // Any App Quality configuration options + }, + "comment": string + } + ] +} +``` + +### Options + +| Option | Required | Description | +| --------- | -------- | --------------------------------------------------------------------------------------------------------- | +| `name` | Required | The profile name that must match a run tag to activate this profile. | +| `config` | Required | An object containing any App Quality configuration options. These values override the root configuration. | +| `comment` | Optional | A comment describing the purpose of this profile. | + +## Examples + +### Basic profile structure + +#### Config + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false, + "comment": "Exclude temporary test elements" + } + ], + "profiles": [ + { + "name": "aq-config-regression", + "config": { + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false, + "comment": "Exclude temporary test elements in regression runs" + }, + { + "selector": "[data-role='debug']", + "include": false, + "comment": "Exclude debug elements in regression runs" + } + ] + } + } + ] +} +``` + +#### Usage + +To use this profile, run Cypress with a matching tag: + +```shell +cypress run --record --tag "aq-config-regression" +``` + +When this tag is used, the profile's `elementFilters` configuration will override the base `elementFilters`, excluding both temporary test elements and debug elements. + +--- + +### Multiple profiles + +You can define multiple profiles for different scenarios: + +#### Config + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false + } + ], + "profiles": [ + { + "name": "aq-config-regression", + "config": { + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false + }, + { + "selector": "[data-role='debug']", + "include": false, + "comment": "Exclude debug elements in regression runs" + } + ] + } + }, + { + "name": "aq-config-staging", + "config": { + "viewFilters": [ + { + "pattern": "https://staging.example.com/admin/*", + "include": false, + "comment": "Exclude admin pages in staging runs" + } + ] + } + } + ] +} +``` + +#### Usage + +Use different tags to activate different profiles: + +```shell +# For regression runs +cypress run --record --tag "aq-config-regression" + +# For staging runs +cypress run --record --tag "aq-config-staging" +``` + +--- + +### Profile with nested configuration + +Profiles can override configuration at any level, including nested configuration specific to Cypress Accessibility or UI Coverage, if your project has both projects enabled. For example: + +#### Config + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false + } + ], + "uiCoverage": { + "attributeFilters": [ + { + "attribute": "id", + "value": ":r.*:", + "include": false, + "comment": "Filter out React auto-generated IDs" + } + ] + }, + "profiles": [ + { + "name": "aq-config-feature-branch", + "config": { + "uiCoverage": { + "elementGroups": [ + { + "selector": "[data-feature='new-checkout']", + "name": "New Checkout Flow", + "comment": "Group new checkout elements for feature branch testing" + } + ] + } + } + } + ] +} +``` + +#### Usage + +```shell +cypress run --record --tag "aq-config-feature-branch" +``` + +This profile adds element grouping for the new checkout flow while keeping the base configuration for element filters and attribute filters. + +--- + +### Profile selection with multiple matching tags + +If you pass multiple tags and more than one matches a profile name, the first matching profile in the `profiles` array is used: + +#### Config + +```json +{ + "profiles": [ + { + "name": "aq-config-regression", + "config": { + "elementFilters": [ + { + "selector": "[data-role='debug']", + "include": false + } + ] + } + }, + { + "name": "aq-config-staging", + "config": { + "viewFilters": [ + { + "pattern": "https://staging.example.com/admin/*", + "include": false + } + ] + } + } + ] +} +``` + +#### Usage + +```shell +cypress run --record --tag "aq-config-regression,aq-config-staging" +``` + +In this case, the `aq-config-regression` profile will be used because it appears first in the `profiles` array, even though both tags match profile names. diff --git a/docs/partials/_viewfilters.mdx b/docs/partials/_viewfilters.mdx index d677361e85..a6c22304d9 100644 --- a/docs/partials/_viewfilters.mdx +++ b/docs/partials/_viewfilters.mdx @@ -27,7 +27,8 @@ supported, if you need to split them up. "viewFilters": [ { "pattern": string, - "include": boolean + "include": boolean, + "comment": string } ] } @@ -41,6 +42,7 @@ For every URL visited and link element found, the first `viewFilters` rule for w | --------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------- | | `pattern` | Required | | A string that matches URLs using [URL Pattern API](https://developer.mozilla.org/en-US/docs/Web/API/URL_Pattern_API) syntax. | | `include` | Optional | `true` | A boolean that determines whether matching URLs should be included in the report. | +| `comment` | Optional | | A comment describing the purpose of this filter rule. | ## Examples diff --git a/docs/partials/_views.mdx b/docs/partials/_views.mdx index b458411835..bc453d4d6b 100644 --- a/docs/partials/_views.mdx +++ b/docs/partials/_views.mdx @@ -34,7 +34,8 @@ URLs with the same values for the specified parameters will be grouped together, "pattern": string, "groupBy": [ string - ] + ], + "comment": string } ] } @@ -48,6 +49,7 @@ The first pattern that a given URL matches is used as its view. If a URL doesn't | --------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pattern` | Required | | A URL pattern to group matching URLs into a single view. Uses [URL Pattern API](https://developer.mozilla.org/en-US/docs/Web/API/URL_Pattern_API) syntax. | | `groupBy` | Optional | | An array of named parameters from your pattern that should create separate views. | +| `comment` | Optional | | A comment describing the purpose of this view configuration. | ## Examples diff --git a/docs/ui-coverage/configuration/allowedinteractioncommands.mdx b/docs/ui-coverage/configuration/allowedinteractioncommands.mdx index 98264e2f36..68933d8b1b 100644 --- a/docs/ui-coverage/configuration/allowedinteractioncommands.mdx +++ b/docs/ui-coverage/configuration/allowedinteractioncommands.mdx @@ -26,7 +26,8 @@ This is particularly useful for filtering out irrelevant interactions or focusin "allowedInteractionCommands": [ { "selector": string, - "commands": [string] + "commands": [string], + "comment": string } ] } @@ -41,6 +42,7 @@ The `allowedInteractionCommands` property accepts an array of objects, where eac | ---------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `selector` | Required | | A CSS selector to identify elements. Supports standard CSS selector syntax, including IDs, classes, attributes, and combinators. | | `commands` | Required | | An array of command names (strings) that should be tracked as interactions for elements matching the selector. All other interaction commands will be ignored for these elements. | +| `comment` | Optional | | A comment describing the purpose of this allowed interaction commands configuration. | ## Examples diff --git a/docs/ui-coverage/configuration/elementgroups.mdx b/docs/ui-coverage/configuration/elementgroups.mdx index 812fde8ed6..980c6f7a62 100644 --- a/docs/ui-coverage/configuration/elementgroups.mdx +++ b/docs/ui-coverage/configuration/elementgroups.mdx @@ -26,7 +26,8 @@ UI Coverage provides logic to automatically [group](/ui-coverage/core-concepts/e "elementGroups": [ { "selector": string, - "name": string + "name": string, + "comment": string } ] } @@ -41,6 +42,7 @@ For every element considered by UI Coverage, the first `elementGroup` rule for w | ---------- | -------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------- | | `selector` | Required | | A CSS selector to identify elements. Supports standard CSS selector syntax, including IDs, classes, attributes, and combinators. | | `name` | Optional | `selector` | A human-readable name for the group, displayed in UI Coverage reports. | +| `comment` | Optional | | A comment describing the purpose of this element group configuration. | ## Examples diff --git a/docs/ui-coverage/configuration/elements.mdx b/docs/ui-coverage/configuration/elements.mdx index d768403c03..c6baa57543 100644 --- a/docs/ui-coverage/configuration/elements.mdx +++ b/docs/ui-coverage/configuration/elements.mdx @@ -27,7 +27,8 @@ The `elements` configuration is used as the element's identity if **only one ele "elements": [ { "selector": string, - "name": string + "name": string, + "comment": string } ] } @@ -44,6 +45,7 @@ If multiple elements within the same snapshot satisfy the same rule, the rule ca | ---------- | -------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------- | | `selector` | Required | | A CSS selector to identify elements. Supports standard CSS selector syntax, including IDs, classes, attributes, and combinators. | | `name` | Optional | `selector` | A human-readable name for the element, displayed in UI Coverage reports. | +| `comment` | Optional | | A comment describing the purpose of this element configuration. | ## Examples diff --git a/docs/ui-coverage/configuration/overview.mdx b/docs/ui-coverage/configuration/overview.mdx index c4fe1903f9..964d113268 100644 --- a/docs/ui-coverage/configuration/overview.mdx +++ b/docs/ui-coverage/configuration/overview.mdx @@ -44,6 +44,28 @@ For a quick overview of the practical application of the most common UI Coverage ::: +### Comments + +All configuration objects support an optional `comment` property that you can use to provide context and explanations for why certain values are set. This helps make your configuration easier to understand and maintain, especially when working in teams or revisiting configuration after some time. + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false, + "comment": "Exclude temporary test elements from coverage" + } + ] +} +``` + +### Profiles + +The `profiles` property allows you to use different configuration settings for different runs based on [run tags](/app/references/command-line#cypress-run-tag-lt-tag-gt). See the [Profiles](/ui-coverage/configuration/profiles) guide for more details. + +### Complete configuration example + A complete configuration with all available options looks as follows: ```json @@ -51,7 +73,8 @@ A complete configuration with all available options looks as follows: "elementFilters": [ { "selector": string, - "include": boolean + "include": boolean, + "comment": string } ], "views": [ @@ -59,13 +82,23 @@ A complete configuration with all available options looks as follows: "pattern": string, "groupBy": [ string - ] + ], + "comment": string } ], "viewFilters": [ { "pattern": string, - "include": boolean + "include": boolean, + "comment": string + } + ], + "profiles": [ + { + "name": string, + "config": { + // Any App Quality configuration options + } } ], "uiCoverage": { @@ -73,13 +106,15 @@ A complete configuration with all available options looks as follows: { "attribute": string, "value": string, - "include": boolean + "include": boolean, + "comment": string } ], "elementGroups": [ { "selector": string, - "name": string + "name": string, + "comment": string } ], "significantAttributes": [ @@ -112,6 +147,7 @@ Check out the following configuration guides. - [Element Filters](/ui-coverage/configuration/elementfilters) - [Element Groups](/ui-coverage/configuration/elementgroups) - [Elements](/ui-coverage/configuration/elements) +- [Profiles](/ui-coverage/configuration/profiles) - [Significant Attributes](/ui-coverage/configuration/significantattributes) - [View Filters](/ui-coverage/configuration/viewfilters) - [Views](/ui-coverage/configuration/views) diff --git a/docs/ui-coverage/configuration/profiles.mdx b/docs/ui-coverage/configuration/profiles.mdx new file mode 100644 index 0000000000..a2da5d87cc --- /dev/null +++ b/docs/ui-coverage/configuration/profiles.mdx @@ -0,0 +1,12 @@ +--- +title: 'Profiles | Cypress UI Coverage' +description: 'The `profiles` configuration property allows you to create configuration overrides that are applied based on run tags.' +sidebar_label: profiles +sidebar_position: 100 +--- + + + +# profiles + + diff --git a/src/theme/MDXComponents.js b/src/theme/MDXComponents.js index 9620dad1a4..9d0e073a9a 100644 --- a/src/theme/MDXComponents.js +++ b/src/theme/MDXComponents.js @@ -28,6 +28,7 @@ import Icon from "@site/src/components/icon"; import ImportMountFunctions from "@site/docs/partials/_import-mount-functions.mdx"; import IntellisenseCodeCompletion from "@site/docs/partials/_intellisense-code-completion.mdx"; import ProductHeading from "@site/src/components/product-heading"; +import Profiles from "@site/docs/partials/_profiles.mdx"; import SignificantAttributes from "@site/docs/partials/_significantattributes.mdx"; import SourceMaps from "@site/docs/partials/_source-maps.mdx"; import SupportFileConfiguration from "@site/docs/partials/_support-file-configuration.mdx"; @@ -192,6 +193,7 @@ export default { ImportMountFunctions, IntellisenseCodeCompletion, ProductHeading, + Profiles, SignificantAttributes, SourceMaps, SupportFileConfiguration, From 7b28b7a4696c1745226be69a2d1c3eef10742ab7 Mon Sep 17 00:00:00 2001 From: Mark Noonan Date: Tue, 9 Dec 2025 09:27:27 -0500 Subject: [PATCH 2/3] Update docs/partials/_profiles.mdx Co-authored-by: Tyler Biethman --- docs/partials/_profiles.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/partials/_profiles.mdx b/docs/partials/_profiles.mdx index 50e379dab6..9f7e84dbff 100644 --- a/docs/partials/_profiles.mdx +++ b/docs/partials/_profiles.mdx @@ -16,7 +16,7 @@ If more than one tag provided to a run matches a profile in your App Quality pro Use a naming convention like `aq-config-x` (e.g., `aq-config-regression`, `aq-config-staging`) to make it clear that a tag is used for configuration lookup purposes. -While relying on existing tags works just fine, but being explicit will help avoid accidental removal or changes of important tags. +While relying on existing tags works just fine, being explicit will help avoid accidental removal of or changes to important tags. ## Syntax From b9f24514f94a2b5c046a2cd6d659aae08166a525 Mon Sep 17 00:00:00 2001 From: marktnoonan Date: Tue, 9 Dec 2025 10:56:10 -0500 Subject: [PATCH 3/3] wip baseline docs --- .../guides/block-pull-requests.mdx | 254 +++++++++++++++++ docs/accessibility/results-api.mdx | 4 + .../guides/block-pull-requests.mdx | 262 ++++++++++++++++++ docs/ui-coverage/results-api.mdx | 25 ++ 4 files changed, 545 insertions(+) create mode 100644 docs/ui-coverage/guides/block-pull-requests.mdx diff --git a/docs/accessibility/guides/block-pull-requests.mdx b/docs/accessibility/guides/block-pull-requests.mdx index 54e23ea762..2a7ccf24ce 100644 --- a/docs/accessibility/guides/block-pull-requests.mdx +++ b/docs/accessibility/guides/block-pull-requests.mdx @@ -63,3 +63,257 @@ getAccessibilityResults().then((results) => { ``` By examining the results and customizing your response, you gain maximum control over how to handle accessibility violations. Leverage CI environment context, such as tags, to fine-tune responses to specific accessibility outcomes. + +## Comparing against a baseline + +Comparing current results against a stored baseline allows you to detect only new violations that have been introduced, while ignoring existing known issues. This approach is more sophisticated than simply maintaining a list of known failing rules, as it tracks violations at the view level and can detect both regressions (new violations) and improvements (resolved violations). + +This is particularly useful in CI/CD pipelines where you want to fail builds only when new accessibility violations are introduced, allowing you to address existing issues incrementally without blocking deployments. + +### Baseline structure + +You can use any format you like as baseline for comparing reports. In the example code below we generate a baseline in a simplified format, which captures the state of accessibility violations from a specific run. The Results API handler code logs this for every run so that it can be easily copied as a new reference point. + +It includes: + +- **runNumber**: The run number used as the baseline reference +- **severityLevels**: The severity levels to track (e.g., `['critical', 'serious', 'moderate', 'minor']`) +- **views**: An object mapping view display names to arrays of failed rule names for that view + +```javascript +{ + "runNumber": "111", + "severityLevels": [ + "critical", + "serious", + "moderate", + "minor" + ], + "views": { + "/": [ + "aria-dialog-name", + "heading-order", + "scrollable-region-focusable" + ], + "/authorizations": [ + "heading-order", + "listitem", + "region" + ] + } +} +``` + +### Complete example + +The following example demonstrates how to compare current results against a baseline, detect new violations, identify resolved violations, and generate a new baseline when changes are detected. + +```javascript title="scripts/compareAccessibilityBaseline.js" +require('dotenv').config(); + +const { getAccessibilityResults } = require('@cypress/extract-cloud-results'); +const fs = require('fs'); + +const TARGET_SEVERITY_LEVELS = ['critical', 'serious', 'moderate', 'minor']; + +// Parse the run number from an accessibility report URL +const parseRunNumber = (url) => { + return url.split('runs/')[1].split('/accessibility')[0]; +}; + +// Define your baseline - this should be stored and updated as your application improves +const baseline = { + "runNumber": "111", + "severityLevels": [ + "critical", + "serious", + "moderate", + "minor" + ], + "views": { + "/": [ + "aria-dialog-name", + "heading-order", + "scrollable-region-focusable", + "empty-table-header", + "listitem", + "region", + "aria-required-children", + "presentation-role-conflict", + "scope-attr-valid", + "aria-prohibited-attr", + "aria-allowed-attr" + ], + "/authorizations": [ + "heading-order", + "listitem", + "region", + "presentation-role-conflict", + "svg-img-alt", + "aria-allowed-attr" + ] + } +}; + +getAccessibilityResults().then((results) => { + // Create objects to store the results + const viewRules = {}; + const viewsWithNewFailedRules = {}; + const viewsWithMissingRules = {}; + + // Iterate through each view in current results + results.views.forEach(view => { + const displayName = view.displayName; + const ruleNames = view.rules.map(rule => rule.name); + + // Add to our results object + viewRules[displayName] = ruleNames; + + // Check for new failing rules + if (view.rules.length) { + view.rules.forEach(rule => { + if (!TARGET_SEVERITY_LEVELS.includes(rule.severity)) { + return; + } + + // If the view exists in baseline and the rule is not in baseline's failed rules + if (!baseline.views?.[displayName]?.includes(rule.name)) { + if (viewsWithNewFailedRules[displayName]) { + viewsWithNewFailedRules[displayName].newFailedRules.push({ + name: rule.name, + url: rule.accessibilityReportUrl, + }); + } else { + viewsWithNewFailedRules[displayName] = { + newFailedRules: [{ + name: rule.name, + url: rule.accessibilityReportUrl, + }] + }; + } + } + }); + } + }); + + // Check for rules in baseline that are not in current results (resolved rules) + Object.entries(baseline.views).forEach(([displayName, baselineRules]) => { + const currentRules = viewRules[displayName] || []; + const resolvedRules = baselineRules.filter(rule => !currentRules.includes(rule)); + + if (resolvedRules.length > 0) { + viewsWithMissingRules[displayName] = { resolvedRules }; + } + }); + + // Report any missing rules + const countOfViewsWithMissingRules = Object.keys(viewsWithMissingRules).length; + const countOfViewsWithNewFailedRules = Object.keys(viewsWithNewFailedRules).length; + + if (countOfViewsWithMissingRules || countOfViewsWithNewFailedRules) { + // Generate and log the new baseline values if there has been a change + const newBaseline = generateBaseline(results); + console.log('\nTo use this run as the new baseline, copy these values:'); + console.log(JSON.stringify(newBaseline, null, 2)); + fs.writeFileSync('new-baseline.json', JSON.stringify(newBaseline, null, 2)); + } + + if (countOfViewsWithMissingRules) { + console.log('\nThe following Views had rules in the baseline that are no longer failing. This may be due to improvements in these rules, or because you did not run as many tests in this run as the baseline run:'); + console.dir(viewsWithMissingRules, { depth: 3 }); + } else if (!countOfViewsWithNewFailedRules) { + console.log('\nNo new or resolved rules were detected. All violations match the baseline.\n'); + } + + if (countOfViewsWithNewFailedRules) { + // Report any new failing rules + console.error('\nThe following Views had rules violated that were previously passing:'); + console.dir(viewsWithNewFailedRules, { depth: 3 }); + throw new Error( + `${countOfViewsWithNewFailedRules} Views contained new failing accessibility rules.` + ); + } + + return viewRules; +}); + +function generateBaseline(results) { + try { + // Create an object to store the results + const viewRules = {}; + + // Iterate through each view + results.views.forEach(view => { + // Get the displayName and extract the rule names + const displayName = view.displayName; + const ruleNames = view.rules + .filter(rule => TARGET_SEVERITY_LEVELS.includes(rule.severity)) + .map(rule => rule.name); + + // Add to our results object + viewRules[displayName] = ruleNames; + }); + + const runNumber = parseRunNumber(results.views[0].accessibilityReportUrl); + + return { + runNumber, + severityLevels: TARGET_SEVERITY_LEVELS, + views: viewRules + }; + } catch (error) { + console.error('Error parsing accessibility results:', error); + return null; + } +} +``` + +### Key concepts + +#### New failing rules + +A **new failing rule** is a rule that has violations in the current run but was not present in the baseline for that view. These represent regressions that need to be addressed. The script will fail the build if any new failing rules are detected. + +#### Resolved rules + +A **resolved rule** is a rule that was present in the baseline but no longer has violations in the current run. These represent improvements in accessibility. The script reports these but does not fail the build, allowing you to track progress. + +#### Severity filtering + +The baseline comparison only tracks rules at the severity levels you specify in `TARGET_SEVERITY_LEVELS`. This allows you to focus on the severity levels that matter most to your team. Rules at other severity levels are ignored during comparison. + +#### View-level comparison + +Violations are tracked per view (URL pattern or component), allowing you to see exactly which pages or components have regressed or improved. This granular tracking makes it easier to identify the source of changes and assign fixes to the appropriate teams. + +### Best practices + +#### When to update the baseline + +Update your baseline when: +- You've fixed accessibility violations and want to prevent regressions +- You've accepted certain violations as known issues that won't block deployments +- You want to track improvements over time + +Store the baseline in version control so it's versioned alongside your code and accessible in CI environments. + +#### Handling partial reports + +If a run is cancelled or incomplete, the Results API may return a partial report. Consider checking `summary.isPartialReport` before comparing against the baseline, as partial reports may not include all views and could produce false positives. + +#### Managing baseline across branches + +You may want different baselines for different branches (e.g., `main` vs feature branches). Consider storing baselines in branch-specific files or using environment variables to specify which baseline to use. + +#### Storing the baseline + +Common approaches for storing baselines: +- **Version control**: Commit the baseline JSON file to your repository +- **CI artifacts**: Store baselines as build artifacts that can be retrieved in subsequent runs +- **External storage**: Use cloud storage or a database for baselines if you need more sophisticated versioning + +:::info + +This baseline comparison approach is more sophisticated than the simple rule list approach shown earlier in this guide, as it tracks violations at the view level and can detect both regressions and improvements. For simpler use cases, the rule list approach may be sufficient. + +::: diff --git a/docs/accessibility/results-api.mdx b/docs/accessibility/results-api.mdx index c230ac672c..c160ebe7d9 100644 --- a/docs/accessibility/results-api.mdx +++ b/docs/accessibility/results-api.mdx @@ -257,6 +257,10 @@ The Accessibility results for the run are returned as an object containing the f } ``` +## Comparing against a baseline + +For comprehensive examples of comparing results against a baseline, including complete code examples, baseline structure, and best practices, see the [Block pull requests and set policies](/accessibility/guides/block-pull-requests#comparing-against-a-baseline) guide. + ### **2. Add to CI Workflow** In your CI workflow that runs your Cypress tests, diff --git a/docs/ui-coverage/guides/block-pull-requests.mdx b/docs/ui-coverage/guides/block-pull-requests.mdx new file mode 100644 index 0000000000..f13bc2a48a --- /dev/null +++ b/docs/ui-coverage/guides/block-pull-requests.mdx @@ -0,0 +1,262 @@ +--- +sidebar_label: Block pull requests and set policies +title: 'Block pull requests and set policies | Cypress UI Coverage Documentation' +description: "Set policies and block pull requests automatically with Cypress UI Coverage's Results API, enabling custom CI workflows to enforce test coverage standards and prevent regressions." +sidebar_position: 40 +--- + + + +# Block pull requests and set policies + +Cypress UI Coverage reports are generated server-side in Cypress Cloud, based on test artifacts uploaded during execution. This ensures there is no performance impact on your Cypress test runs. + +## Using the Results API + +The [Cypress UI Coverage Results API](/ui-coverage/results-api) allows you to access UI Coverage results post-test run, enabling workflows like blocking pull requests or triggering alerts based on specific coverage criteria. This involves adding a dedicated UI Coverage verification step to your CI pipeline. With a Cypress helper function, you can automatically fetch the report for the relevant test run within the CI build context. + +## Implementing a status check + +The Results API offers full flexibility to analyze results and take tailored actions. It can also integrate with status checks on pull requests, allowing you to block merges when coverage thresholds are not met. + +## Defining policies in the verification step + +The [Results API Documentation](/ui-coverage/results-api) provides detailed guidance on the API's capabilities. Here's a simplified example demonstrating how to enforce a minimum coverage threshold: + +```js +const { getUICoverageResults } = require('@cypress/extract-cloud-results') + +// Fetch UI Coverage results +getUICoverageResults().then((results) => { + const { summary, views } = results + + // Verify overall coverage + if (summary.coverage < 80) { + throw new Error( + `Project coverage is ${summary.coverage}%, below the minimum threshold of 80%.` + ) + } + + // Verify critical view coverage + const criticalViews = [/login/, /checkout/] + views.forEach((view) => { + if ( + criticalViews.some((pattern) => pattern.test(view.displayName)) && + view.coverage < 95 + ) { + throw new Error( + `Critical view "${view.displayName}" coverage is ${view.coverage}%, below the required 95%.` + ) + } + }) +}) +``` + +By examining the results and customizing your response, you gain maximum control over how to handle coverage gaps. Leverage CI environment context, such as tags, to fine-tune responses to specific coverage outcomes. + +## Comparing against a baseline + +Comparing current results against a stored baseline allows you to detect only new untested elements that have been introduced, while ignoring existing coverage gaps. This approach helps you focus on regressions in test coverage and track improvements over time. + +This is particularly useful in CI/CD pipelines where you want to fail builds only when new untested elements are introduced, allowing you to address existing coverage gaps incrementally without blocking deployments. + +### Baseline structure + +A baseline is a JSON object that captures the state of untested elements from a specific run. It includes: + +- **runNumber**: The run number used as the baseline reference +- **views**: An object mapping view display names to arrays of view identifiers that had untested elements + +Since the UI Coverage Results API provides `untestedElementsCount` per view rather than individual element identifiers, the baseline tracks which views have untested elements (where `untestedElementsCount > 0`). + +```javascript +{ + "runNumber": "111", + "views": { + "/": [], + "/authorizations": [], + "/checkout": ["/checkout"], + "/profile": ["/profile"] + } +} +``` + +In this example, `/checkout` and `/profile` had untested elements in the baseline run, while `/` and `/authorizations` had full coverage. + +### Complete example + +The following example demonstrates how to compare current results against a baseline, detect new views with untested elements, identify views where coverage has improved, and generate a new baseline when changes are detected. + +```javascript title="scripts/compareUICoverageBaseline.js" +require('dotenv').config(); + +const { getUICoverageResults } = require('@cypress/extract-cloud-results'); +const fs = require('fs'); + +// Parse the run number from a UI Coverage report URL +const parseRunNumber = (url) => { + return url.split('runs/')[1].split('/ui-coverage')[0]; +}; + +// Define your baseline - this should be stored and updated as your test coverage improves +const baseline = { + "runNumber": "111", + "views": { + "/": [], + "/authorizations": [], + "/checkout": ["/checkout"], + "/profile": ["/profile"] + } +}; + +getUICoverageResults().then((results) => { + // Create objects to store the results + const viewUntestedElements = {}; + const viewsWithNewUntestedElements = {}; + const viewsWithResolvedUntestedElements = {}; + + // Iterate through each view in current results + results.views.forEach(view => { + const displayName = view.displayName; + const hasUntestedElements = view.untestedElementsCount > 0; + + // Track which views have untested elements + if (hasUntestedElements) { + viewUntestedElements[displayName] = [displayName]; + } else { + viewUntestedElements[displayName] = []; + } + + // Check for new views with untested elements + // A view has new untested elements if: + // 1. It has untested elements in current run, AND + // 2. It either didn't exist in baseline OR had no untested elements in baseline + if (hasUntestedElements) { + const baselineHadUntested = baseline.views?.[displayName]?.length > 0; + + if (!baselineHadUntested) { + viewsWithNewUntestedElements[displayName] = { + untestedElementsCount: view.untestedElementsCount, + url: view.uiCoverageReportUrl + }; + } + } + }); + + // Check for views in baseline that no longer have untested elements (resolved) + Object.entries(baseline.views).forEach(([displayName, baselineUntested]) => { + const currentHasUntested = viewUntestedElements[displayName]?.length > 0; + const baselineHadUntested = baselineUntested.length > 0; + + // If baseline had untested elements but current run doesn't, it's resolved + if (baselineHadUntested && !currentHasUntested) { + viewsWithResolvedUntestedElements[displayName] = { + resolved: true + }; + } + }); + + // Report any changes + const countOfViewsWithResolved = Object.keys(viewsWithResolvedUntestedElements).length; + const countOfViewsWithNewUntested = Object.keys(viewsWithNewUntestedElements).length; + + if (countOfViewsWithResolved || countOfViewsWithNewUntested) { + // Generate and log the new baseline values if there has been a change + const newBaseline = generateBaseline(results); + console.log('\nTo use this run as the new baseline, copy these values:'); + console.log(JSON.stringify(newBaseline, null, 2)); + fs.writeFileSync('new-baseline.json', JSON.stringify(newBaseline, null, 2)); + } + + if (countOfViewsWithResolved) { + console.log('\nThe following Views had untested elements in the baseline that are now fully tested. This indicates improved test coverage:'); + console.dir(viewsWithResolvedUntestedElements, { depth: 3 }); + } else if (!countOfViewsWithNewUntested) { + console.log('\nNo new or resolved untested elements were detected. All coverage matches the baseline.\n'); + } + + if (countOfViewsWithNewUntested) { + // Report any new untested elements + console.error('\nThe following Views have new untested elements that were not present in the baseline:'); + console.dir(viewsWithNewUntestedElements, { depth: 3 }); + throw new Error( + `${countOfViewsWithNewUntested} Views contained new untested elements.` + ); + } + + return viewUntestedElements; +}); + +function generateBaseline(results) { + try { + // Create an object to store the results + const viewUntestedElements = {}; + + // Iterate through each view + results.views.forEach(view => { + const displayName = view.displayName; + const hasUntestedElements = view.untestedElementsCount > 0; + + // Track views that have untested elements + // Store the view name in an array if it has untested elements, empty array otherwise + viewUntestedElements[displayName] = hasUntestedElements ? [displayName] : []; + }); + + const runNumber = parseRunNumber(results.views[0].uiCoverageReportUrl); + + return { + runNumber, + views: viewUntestedElements + }; + } catch (error) { + console.error('Error parsing UI Coverage results:', error); + return null; + } +} +``` + +### Key concepts + +#### New untested elements + +A **new untested element** situation occurs when a view has untested elements in the current run but did not have untested elements in the baseline. This represents a regression in test coverage that needs to be addressed. The script will fail the build if any views with new untested elements are detected. + +#### Resolved untested elements + +A **resolved untested element** situation occurs when a view had untested elements in the baseline but no longer has untested elements in the current run. This represents an improvement in test coverage. The script reports these but does not fail the build, allowing you to track progress. + +#### View-level comparison + +Untested elements are tracked per view (URL pattern or component), allowing you to see exactly which pages or components have coverage regressions or improvements. This granular tracking makes it easier to identify where new tests are needed or where coverage has improved. + +### Best practices + +#### When to update the baseline + +Update your baseline when: +- You've added tests to cover previously untested elements and want to prevent regressions +- You've accepted certain coverage gaps as known issues that won't block deployments +- You want to track coverage improvements over time + +Store the baseline in version control so it's versioned alongside your code and accessible in CI environments. + +#### Handling partial reports + +If a run is cancelled or incomplete, the Results API may return a partial report. Consider checking `summary.isPartialReport` before comparing against the baseline, as partial reports may not include all views and could produce false positives. + +#### Managing baseline across branches + +You may want different baselines for different branches (e.g., `main` vs feature branches). Consider storing baselines in branch-specific files or using environment variables to specify which baseline to use. + +#### Storing the baseline + +Common approaches for storing baselines: +- **Version control**: Commit the baseline JSON file to your repository +- **CI artifacts**: Store baselines as build artifacts that can be retrieved in subsequent runs +- **External storage**: Use cloud storage or a database for baselines if you need more sophisticated versioning + +:::info + +This baseline comparison approach complements the [Branch Review](/ui-coverage/guides/compare-reports) UI feature, which provides visual comparisons between runs. The programmatic approach is ideal for CI/CD automation, while Branch Review is better suited for manual investigation and code review workflows. + +::: diff --git a/docs/ui-coverage/results-api.mdx b/docs/ui-coverage/results-api.mdx index a789bc177c..b97872dac2 100644 --- a/docs/ui-coverage/results-api.mdx +++ b/docs/ui-coverage/results-api.mdx @@ -47,6 +47,27 @@ If you check this in as a dependency, your installation will fail when we update Write a script to fetch UI Coverage results and assert test coverage criteria. This script will be executed in CI. +#### Basic example + +This snippet uses the `getUICoverageResults()` helper to log out the results. It assumes your Project ID and Record Key variable are set. The following should work in any of the supported CI Providers out of the box: + +```javascript title="scripts/verifyUICoverageResults.js" +// Assuming these environment variables are set: +// CYPRESS_PROJECT_ID=your-id +// CYPRESS_RECORD_KEY=your-record-key + +const { getUICoverageResults } = require('@cypress/extract-cloud-results') + +getUICoverageResults().then((results) => { + // use `console.dir` instead of `console.log` because the data is nested + console.dir(results, { depth: Infinity }) +}) +``` + +#### How to assert test coverage meets your requirements + +The following example demonstrates how to verify that test coverage meets minimum thresholds: + ```javascript title="scripts/verifyUICoverageResults.js" const { getUICoverageResults } = require('@cypress/extract-cloud-results') @@ -152,6 +173,10 @@ The `getUICoverageResults` utility returns the following data: } ``` +## Comparing against a baseline + +For comprehensive examples of comparing results against a baseline, including complete code examples, baseline structure, and best practices, see the [Block pull requests and set policies](/ui-coverage/guides/block-pull-requests#comparing-against-a-baseline) guide. + ### **2. Add to CI Workflow** In your CI workflow that runs your Cypress tests,