-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[SDTEST-3123] Update Flaky Tests Management docs #33185
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Preview links (active after the
|
estherk15
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a suggestion to fix the list formatting, but approved!
| | ---- | ---- | | ||
| | **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is `Active` for a specified number of days. <li>Branch: Quarantine an `Active` test if it flakes in one or more specified branches.| | ||
| | **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. <li>Branch: Disable an `Active` or `Quarantined` test if it flakes in one or more specified branches. | | ||
| | **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is `Active` for a specified number of days. The rule is triggered every day at 12:15 UTC. <li>Branch: Quarantine an `Active` test if it flakes in one or more specified branches. <li>Failure rate: Quarantine an `Active` test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes. | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| | **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is `Active` for a specified number of days. The rule is triggered every day at 12:15 UTC. <li>Branch: Quarantine an `Active` test if it flakes in one or more specified branches. <li>Failure rate: Quarantine an `Active` test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes. | | |
| | **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <ul><li>Time: Quarantine a test if its status is `Active` for a specified number of days. The rule is triggered every day at 12:15 UTC. <li>Branch: Quarantine an `Active` test if it flakes in one or more specified branches. <li>Failure rate: Quarantine an `Active` test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes.</ul> | |
| | **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is `Active` for a specified number of days. <li>Branch: Quarantine an `Active` test if it flakes in one or more specified branches.| | ||
| | **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. <li>Branch: Disable an `Active` or `Quarantined` test if it flakes in one or more specified branches. | | ||
| | **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is `Active` for a specified number of days. The rule is triggered every day at 12:15 UTC. <li>Branch: Quarantine an `Active` test if it flakes in one or more specified branches. <li>Failure rate: Quarantine an `Active` test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes. | | ||
| | **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. The rule is triggered every day at 12:30 UTC. <li>Branch: Disable an `Active` or `Quarantined` test if it flakes in one or more specified branches. <li>Failure rate: Disable an `Active` or `Quarantined` test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes. | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| | **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. The rule is triggered every day at 12:30 UTC. <li>Branch: Disable an `Active` or `Quarantined` test if it flakes in one or more specified branches. <li>Failure rate: Disable an `Active` or `Quarantined` test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes. | | |
| | **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. The rule is triggered every day at 12:30 UTC. <ul><li>Branch: Disable an `Active` or `Quarantined` test if it flakes in one or more specified branches. <li>Failure rate: Disable an `Active` or `Quarantined` test if its failure rate over the last 7 days is greater or equal to the specified threshold. The rule is triggered every 15 minutes. </ul>| |

What does this PR do? What is the motivation?
Updating the docs of Flaky Tests Management documenting the new "Failure rate" automatic rule.
Updating mentions of "Flaky Test Management" to "Flaky Tests Management" for coherence. In our DataDog UI, the product is called "Flaky Tests Management".

Merge instructions
Merge readiness:
Do not merge after approving.
For Datadog employees:
Your branch name MUST follow the
<name>/<description>convention and include the forward slash (/). Without this format, your pull request will not pass CI, the GitLab pipeline will not run, and you won't get a branch preview. Getting a branch preview makes it easier for us to check any issues with your PR, such as broken links.If your branch doesn't follow this format, rename it or create a new branch and PR.
[6/5/2025] Merge queue has been disabled on the documentation repo. If you have write access to the repo, the PR has been reviewed by a Documentation team member, and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #documentation channel in Slack.
Additional notes