From c40b63db6bcfa9d57068167c8ffdc2d847661b98 Mon Sep 17 00:00:00 2001 From: Rohan Bhaumik Date: Mon, 1 Dec 2025 13:05:47 -0500 Subject: [PATCH 1/2] docs: Clarify GitHub status checks for AI Code Review - Add GitHub Permissions section explaining required permissions (Pull Requests, Checks, Commit Statuses) - Add comprehensive GitHub Status Checks section covering: - Status check behavior (pass/neutral/fail states) - Visibility information for repository users - Instructions for integrating with branch protection rules - Add three new FAQ entries about status checks: - Why status checks fail - How to disable status checks - Update GitHub integration docs to reference AI Code Review in permissions table Fixes: https://linear.app/getsentry/issue/CW-11 --- .../source-code-mgmt/github/index.mdx | 6 +- .../ai-in-sentry/ai-code-review/index.mdx | 62 ++++++++++++++++++- 2 files changed, 64 insertions(+), 4 deletions(-) diff --git a/docs/organization/integrations/source-code-mgmt/github/index.mdx b/docs/organization/integrations/source-code-mgmt/github/index.mdx index 11a6b4b7da810..34f91ab70ec3c 100644 --- a/docs/organization/integrations/source-code-mgmt/github/index.mdx +++ b/docs/organization/integrations/source-code-mgmt/github/index.mdx @@ -65,13 +65,13 @@ The following permissions are required for the GitHub integration to work correc | Permission Category | Access Level | Why Sentry Needs It | |--------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| | Administration | Read-only | To respect branch protection rules, determine default branches, and enhance automation. | -| Checks | Read & Write | To show detailed pass/fail results (e.g. test coverage, release health) as GitHub Checks on PRs or commits. | -| Commit Statuses | Read & Write | To post status checks (e.g. test analytics data) to commits in GitHub and integrate with branch protection rules. | +| Checks | Read & Write | To show detailed pass/fail results (e.g. test coverage, release health, AI Code Review) as GitHub Checks on PRs or commits. | +| Commit Statuses | Read & Write | To post status checks (e.g. test analytics data, AI Code Review) to commits in GitHub and integrate with branch protection rules. | | Contents | Read-only | To fetch source files, commits, and blame data for Suspect Commits, tracking files in Releases. | | Issues | Read & Write | To create GitHub issues from Sentry events and update linked issues. | | Members
_[Org-level]_ | Read-only | To map organization users to GitHub identities for improved issue attribution and collaboration. | | Metadata | Read-only | To identify repository structure, names, default branches, and visibility. | -| Pull Requests | Read & Write | To write comments on pull requests for (1) issues caused by the pull request and (2) to highlight existing issues in the code diff.| +| Pull Requests | Read & Write | To write comments on pull requests for (1) issues caused by the pull request, (2) to highlight existing issues in the code diff, and (3) AI Code Review feedback.| | Webhooks | Read & Write | To subscribe to real-time updates like push events, PRs, and issues. Occasionally, Sentry will request additional permissions to your GitHub account as new features are introduced. Denying or ignoring the request to update will not affect your current Sentry usage, but may prevent access to future features. diff --git a/docs/product/ai-in-sentry/ai-code-review/index.mdx b/docs/product/ai-in-sentry/ai-code-review/index.mdx index e8943a3ab2b6b..b643489ffbfac 100644 --- a/docs/product/ai-in-sentry/ai-code-review/index.mdx +++ b/docs/product/ai-in-sentry/ai-code-review/index.mdx @@ -26,6 +26,20 @@ To enable AI Code Review in your GitHub organization or on specific repositories - `Show Generative AI Features` - `Enable AI Code Review` +### GitHub Permissions + +AI Code Review requires specific GitHub permissions to function properly. When you install or update the Sentry GitHub integration, you'll need to accept the following permissions: + +- **Pull Requests (Read & Write)**: To read PR content and write code review comments +- **Checks (Read & Write)**: To create status checks that show the AI Code Review results on your PRs +- **Commit Statuses (Read & Write)**: To post status checks to commits and integrate with branch protection rules + +If you have previously installed the GitHub integration, you may need to update your permissions to include these. GitHub will prompt you to accept updated permissions when Sentry requests them. + + + Denying or ignoring permission update requests won't affect your current Sentry usage, but will prevent AI Code Review and other features from working. + + ## AI Code Review Commands Ways AI Code Review can help you: @@ -38,6 +52,39 @@ Once you have added a comment, the assistant will reply, acknowledging the reque ![Sentry Prevent AI Overview =900x](./img/prevent-ai-response.png) +## GitHub Status Checks + +AI Code Review creates GitHub status checks (also called "checks") on your pull requests to indicate whether the code review passed or found potential issues. These status checks appear in the PR's checks section and can be integrated with GitHub's branch protection rules. + +### Status Check Behavior + +- **Pass**: If AI Code Review doesn't find any errors or concerns in your code, the status check will pass (green checkmark). +- **Neutral**: If AI Code Review runs successfully but has suggestions or comments, the status check will be neutral (gray) and won't block merging. +- **Fail**: The status check only fails if there's an error running the AI Code Review itself (for example, if the feature is not enabled), not based on code quality. + +### Visibility + +Status checks from AI Code Review are visible to: +- All users with read access to the repository +- Anyone viewing the pull request on GitHub +- GitHub Actions and other CI/CD tools that check PR status + +The detailed code review comments are also visible to all users with repository access, appearing as regular PR comments. + +### Using with Branch Protection + +You can optionally configure GitHub branch protection rules to require the AI Code Review status check before merging. To do this: + +1. Go to your GitHub repository Settings > Branches +2. Add or edit a branch protection rule for your target branch (e.g., `main`) +3. Enable "Require status checks to pass before merging" +4. Search for and select the "Sentry AI Code Review" status check +5. Save your changes + + + Be aware that requiring the AI Code Review status check will block PR merges if the check fails due to configuration issues or service disruptions. We recommend monitoring the feature during beta before making it a required check. + + ## Frequently Asked Questions - **What data does AI Code Review need access to for the AI system to function, and what information is sent to third-party AI providers?** @@ -58,6 +105,19 @@ Once you have added a comment, the assistant will reply, acknowledging the reque Error Prediction is automatically triggered by the following GitHub pull_request webhook events: - `opened`: when you open a new pull request (we skip those opened in `draft` state) - - `ready_for_review`: when a draft pull request is marked as “Ready for review” + - `ready_for_review`: when a draft pull request is marked as "Ready for review" To manually run error prediction and get a general review, comment `@sentry review` in the PR. + +- **Why is the AI Code Review status check failing?** + + The status check may fail if: + - The `Show Generative AI Features` or `Enable AI Code Review` settings are not enabled in your organization + - The GitHub integration doesn't have the required permissions (Pull Requests, Checks, and Commit Statuses) + - There's a temporary service issue + + The status check does not fail based on code quality issues found during review - those appear as comments and a neutral status. + +- **Can I disable status checks while keeping AI Code Review comments?** + + Status checks are automatically created whenever AI Code Review runs. If you don't want them to appear or affect your workflow, simply don't add them as a required check in your branch protection rules. The checks will still appear on PRs but won't block merging. From 6e564bec68d5b0f4bd9175aec24ba3f2455a1d72 Mon Sep 17 00:00:00 2001 From: Rohan Bhaumik Date: Wed, 3 Dec 2025 09:01:44 -0500 Subject: [PATCH 2/2] Address PR review comments - Change 'regular PR comments' to 'review comments' - Fix status check name to 'Seer Code Review' (not 'Sentry AI Code Review') - Update recommendation to NOT make status check required - Update status check behavior to include all states: success, neutral, error, cancelled, timed_out - Clarify that success indicates review completed, not code quality - Rewrite FAQ about failing status checks to be more accurate --- .../ai-in-sentry/ai-code-review/index.mdx | 23 +++++++++---------- 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/docs/product/ai-in-sentry/ai-code-review/index.mdx b/docs/product/ai-in-sentry/ai-code-review/index.mdx index b643489ffbfac..07e76fc76288a 100644 --- a/docs/product/ai-in-sentry/ai-code-review/index.mdx +++ b/docs/product/ai-in-sentry/ai-code-review/index.mdx @@ -58,9 +58,11 @@ AI Code Review creates GitHub status checks (also called "checks") on your pull ### Status Check Behavior -- **Pass**: If AI Code Review doesn't find any errors or concerns in your code, the status check will pass (green checkmark). -- **Neutral**: If AI Code Review runs successfully but has suggestions or comments, the status check will be neutral (gray) and won't block merging. -- **Fail**: The status check only fails if there's an error running the AI Code Review itself (for example, if the feature is not enabled), not based on code quality. +- **Success**: If AI Code Review completes successfully, the status check will show as successful (green checkmark). This happens regardless of whether issues were found - the check indicates the review completed, not that the code is error-free. +- **Neutral**: If AI Code Review runs but has no specific pass/fail outcome. +- **Error**: If there's an error running the AI Code Review itself, such as a service issue. +- **Cancelled**: If the review was cancelled, typically because a new commit was pushed and the previous review was superseded. +- **Timed out**: If the review takes too long to complete. ### Visibility @@ -69,7 +71,7 @@ Status checks from AI Code Review are visible to: - Anyone viewing the pull request on GitHub - GitHub Actions and other CI/CD tools that check PR status -The detailed code review comments are also visible to all users with repository access, appearing as regular PR comments. +The detailed code review comments are also visible to all users with repository access, appearing as review comments on the pull request. ### Using with Branch Protection @@ -78,11 +80,11 @@ You can optionally configure GitHub branch protection rules to require the AI Co 1. Go to your GitHub repository Settings > Branches 2. Add or edit a branch protection rule for your target branch (e.g., `main`) 3. Enable "Require status checks to pass before merging" -4. Search for and select the "Sentry AI Code Review" status check +4. Search for and select the "Seer Code Review" status check 5. Save your changes - Be aware that requiring the AI Code Review status check will block PR merges if the check fails due to configuration issues or service disruptions. We recommend monitoring the feature during beta before making it a required check. + We recommend not making the AI Code Review status check a required check. Requiring it will block PR merges if the check fails due to service disruptions, and may conflict with future personal configuration options that allow users to opt out of code review. ## Frequently Asked Questions @@ -109,14 +111,11 @@ Error Prediction is automatically triggered by the following GitHub pull_request To manually run error prediction and get a general review, comment `@sentry review` in the PR. -- **Why is the AI Code Review status check failing?** +- **Why is the AI Code Review status check showing an error?** - The status check may fail if: - - The `Show Generative AI Features` or `Enable AI Code Review` settings are not enabled in your organization - - The GitHub integration doesn't have the required permissions (Pull Requests, Checks, and Commit Statuses) - - There's a temporary service issue + The status check may show an error if there's a temporary service issue or the review timed out. The status check does not fail based on code quality issues found during review - those appear as review comments on your PR. - The status check does not fail based on code quality issues found during review - those appear as comments and a neutral status. + If you don't see a status check at all, ensure that the `Show Generative AI Features` and `Enable AI Code Review` settings are enabled in your [organization settings](https://sentry.io/orgredirect/settings/:orgslug/), and that your GitHub integration has the required Checks permission. - **Can I disable status checks while keeping AI Code Review comments?**