ci: Readability-score delta Github Action #2884
There are no files selected for viewing
Submodule .claude
updated
from 018bf7 to 11d734
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,202 @@ | ||
| # ------------------------------------------------------------------ | ||
| # Readability delta check (DOCS-2626) | ||
| # ------------------------------------------------------------------ | ||
| # | ||
| # What this workflow does | ||
| # ----------------------- | ||
| # Posts an informational, non-blocking PR comment describing how the PR affects | ||
| # the readability of the English docs it touches. The scoring logic lives in the | ||
| # coreweave/docs-skills submodule (.claude/scripts/_readability.py and | ||
| # _docs_eval_lib.py); scripts/readability/pr_report.py is the wandb/docs glue | ||
| # that diffs base..head, scores each changed .mdx file, and builds the Markdown. | ||
| # | ||
| # This check never fails a PR. Every step exits 0; the worst case is a comment | ||
| # noting that scoring was unavailable. | ||
| # | ||
| # Auth | ||
| # ---- | ||
| # - Main checkout uses the default GITHUB_TOKEN (the workflow runs in the | ||
| # wandb/docs PR context). | ||
| # - The coreweave/docs-skills submodule is private and cross-org, so a separate | ||
| # step initializes it with DOCENGINE_TOKEN (the same x-access-token credential | ||
| # used for the gitsubmodule ecosystem in .github/dependabot.yml). That token | ||
| # needs no wandb/docs scope. | ||
| # - The AI agent comprehension judge calls W&B Inference with the | ||
| # WANDB_DOCS_INFERENCE_API_KEY secret (a W&B key with Inference credits), | ||
| # passed to the scorer as WANDB_API_KEY. When that secret is absent the | ||
| # deterministic textstat delta still runs. | ||
| # | ||
| # Forks | ||
| # ----- | ||
| # Fork PRs have no access to repo secrets, so neither the submodule fetch nor the | ||
| # judge can run. The first step detects a fork and makes the whole job a no-op | ||
| # (posts an Actions notice, exits 0). Forks are uncommon in wandb/docs and | ||
| # coreweave repos cannot use forks at all, so the lost coverage is negligible. | ||
| # ------------------------------------------------------------------ | ||
|
|
||
| name: Readability delta | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: [opened, synchronize, reopened] | ||
| paths: | ||
| - "**/*.mdx" | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: | ||
| group: readability-delta-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
| pull-requests: write | ||
|
|
||
| jobs: | ||
| readability: | ||
| name: Report readability delta | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| # Fork no-op guard: fork PRs cannot reach the private submodule or W&B | ||
| # Inference, so skip the whole check (still reports success). | ||
| - name: Skip on fork PRs | ||
| id: fork | ||
| run: | | ||
| if [ "${{ github.event_name }}" = "pull_request" ] && \ | ||
| [ "${{ github.event.pull_request.head.repo.fork }}" = "true" ]; then | ||
| echo "is_fork=true" >> "$GITHUB_OUTPUT" | ||
| echo "::notice::Readability check skipped: fork PRs have no access to the docs-skills submodule or W&B Inference." | ||
| else | ||
| echo "is_fork=false" >> "$GITHUB_OUTPUT" | ||
| fi | ||
|
|
||
| - name: Check out repository | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 | ||
| with: | ||
| # Need full history to diff base..head. | ||
| fetch-depth: 0 | ||
| submodules: false | ||
|
|
||
| # The docs-skills submodule is a private repo in another org. Initialize it | ||
| # with DOCENGINE_TOKEN so the main checkout token needs no extra scope. | ||
| - name: Check out docs-skills submodule | ||
| # Non-fatal: a transient auth/network failure here must not fail the PR. | ||
| # The report step below detects the missing analyzer and posts the | ||
| # informational fallback comment instead. | ||
| continue-on-error: true | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| env: | ||
| DOCENGINE_TOKEN: ${{ secrets.DOCENGINE_TOKEN }} | ||
| run: | | ||
| git config --global url."https://x-access-token:${DOCENGINE_TOKEN}@github.com/".insteadOf "https://github.com/" | ||
| git submodule update --init --recursive .claude | ||
|
Comment on lines
+90
to
+92
|
||
|
|
||
| - name: Set up Python 3.11 | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 | ||
| with: | ||
| python-version: "3.11" | ||
|
|
||
| - name: Cache pip packages | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6 | ||
| with: | ||
| path: ~/.cache/pip | ||
| key: ${{ runner.os }}-pip-readability-${{ hashFiles('scripts/readability/requirements.txt') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-pip-readability- | ||
| ${{ runner.os }}-pip- | ||
|
|
||
| - name: Install dependencies | ||
| # Non-fatal: if deps fail to install, the report step still runs and | ||
| # falls back to the informational notice rather than failing the PR. | ||
| continue-on-error: true | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| run: pip install -r scripts/readability/requirements.txt | ||
|
Comment on lines
+110
to
+115
Contributor
Author
There was a problem hiding this comment. Fixed in 9c13165 — added
Comment on lines
+110
to
+115
|
||
|
|
||
| # Resolve base and head SHAs. For pull_request, GitHub provides them. For | ||
| # workflow_dispatch, diff the merge-base of the current ref against the | ||
| # default branch. | ||
| - name: Resolve base and head | ||
| id: refs | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| env: | ||
| EVENT_NAME: ${{ github.event_name }} | ||
| PR_BASE: ${{ github.event.pull_request.base.sha }} | ||
| PR_HEAD: ${{ github.event.pull_request.head.sha }} | ||
| DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} | ||
| run: | | ||
| set -euo pipefail | ||
| if [ "$EVENT_NAME" = "pull_request" ]; then | ||
| echo "base=$PR_BASE" >> "$GITHUB_OUTPUT" | ||
| echo "head=$PR_HEAD" >> "$GITHUB_OUTPUT" | ||
| else | ||
| git fetch --no-tags --depth=200 origin "$DEFAULT_BRANCH" || true | ||
| BASE=$(git merge-base "origin/${DEFAULT_BRANCH}" HEAD || git rev-parse HEAD~1) | ||
| echo "base=$BASE" >> "$GITHUB_OUTPUT" | ||
| echo "head=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" | ||
| fi | ||
|
|
||
| - name: Build readability report | ||
| id: report | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| env: | ||
| # Optional: enables the AI agent comprehension judge via W&B Inference. | ||
| # The scorer reads WANDB_API_KEY; the repo secret is named | ||
| # WANDB_DOCS_INFERENCE_API_KEY (a W&B key with Inference credits). | ||
| WANDB_API_KEY: ${{ secrets.WANDB_DOCS_INFERENCE_API_KEY }} | ||
| RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} | ||
| run: | | ||
| set -euo pipefail | ||
| JUDGE_FLAG="" | ||
| if [ -n "${WANDB_API_KEY:-}" ]; then | ||
| JUDGE_FLAG="--judge" | ||
| fi | ||
| # Never fail the PR: capture failures and fall back to a notice. | ||
| if python scripts/readability/pr_report.py \ | ||
| --base "${{ steps.refs.outputs.base }}" \ | ||
| --head "${{ steps.refs.outputs.head }}" \ | ||
| --include-marker \ | ||
| --run-url "${RUN_URL}" \ | ||
| --run-id "${GITHUB_RUN_ID}" \ | ||
| ${JUDGE_FLAG} \ | ||
| > comment-body.md 2> report-errors.log; then | ||
| echo "ok=true" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "ok=false" >> "$GITHUB_OUTPUT" | ||
| echo "::warning::Readability report failed to build; see logs." | ||
| cat report-errors.log || true | ||
| { | ||
| echo "<!-- readability-delta-report -->" | ||
| echo "" | ||
| echo "## Readability impact" | ||
| echo "" | ||
| echo "The readability check could not run for this push. This is informational and does not block the PR." | ||
| } > comment-body.md | ||
| fi | ||
|
|
||
| - name: Upsert readability report on PR | ||
| if: steps.fork.outputs.is_fork != 'true' && github.event_name == 'pull_request' | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| PR_NUMBER: ${{ github.event.pull_request.number }} | ||
| run: | | ||
| MARKER="<!-- readability-delta-report -->" | ||
| COMMENT_ID=$(gh api \ | ||
| "repos/${{ github.repository }}/issues/${PR_NUMBER}/comments" \ | ||
| --paginate -q ".[] | select(.body | contains(\"${MARKER}\")) | .id" \ | ||
| | head -n1) | ||
|
|
||
| if [ -n "${COMMENT_ID}" ]; then | ||
| gh api \ | ||
| "repos/${{ github.repository }}/issues/comments/${COMMENT_ID}" \ | ||
| -X PATCH -F "body=@comment-body.md" | ||
| echo "Updated existing readability report comment ${COMMENT_ID}" | ||
| else | ||
| gh pr comment "${PR_NUMBER}" --body-file comment-body.md | ||
| echo "Created new readability report comment" | ||
| fi | ||
|
|
||
| - name: Write readability report to job summary | ||
| if: steps.fork.outputs.is_fork != 'true' && github.event_name == 'workflow_dispatch' | ||
| run: cat comment-body.md >> "${GITHUB_STEP_SUMMARY}" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Readability delta check | ||
|
|
||
| An informational, non-blocking CI check that reports how a pull request affects | ||
| the readability of the docs it touches (DOCS-2626). It posts a single PR comment | ||
| with a word-weighted Flesch-Kincaid grade delta across the changed pages, plus an | ||
| optional AI-agent-comprehension rating. | ||
|
|
||
| We report the *delta*, not an absolute grade. Technical docs have an inherently | ||
| high reading grade level, so "this page went from 12.4 to 11.1" is meaningful | ||
| where "12.4" on its own is not. A lower Flesch-Kincaid grade and a higher Flesch | ||
| reading ease both mean easier to read. | ||
|
|
||
| ## How it works | ||
|
|
||
| The scoring logic lives in the `coreweave/docs-skills` submodule (`.claude/`), | ||
| shared with `coreweave/documentation`: | ||
|
|
||
| - `.claude/scripts/_readability.py` extracts narrative prose from MDX (stripping | ||
| frontmatter, code, JSX/Mintlify components, tables, and links) and scores it | ||
| with `textstat` (Flesch reading ease, Flesch-Kincaid grade, Gunning fog, SMOG). | ||
| - `.claude/scripts/_docs_eval_lib.py` provides the AI-agent-comprehension LLM | ||
| judge (W&B Inference, authenticated with `WANDB_API_KEY`). | ||
| - `.claude/scripts/readability_baselines.json` holds per-doc-type baseline grade | ||
| levels (regenerate with `.claude/scripts/readability_baselines.py`). | ||
|
|
||
| This directory holds the wandb/docs plumbing: | ||
|
|
||
| - `pr_report.py` diffs the PR base and head, scores each changed English `.mdx` | ||
| file, aggregates a word-weighted delta, and builds the Markdown report. | ||
| - `tests/` covers the parsing and Markdown-building logic (no network needed). | ||
|
|
||
| The workflow is `.github/workflows/readability-delta.yml`. | ||
|
|
||
| ## Behavior | ||
|
|
||
| - **Non-blocking**: the check never fails a PR. It only posts a comment. | ||
| - **Forks**: skipped (no access to the submodule or W&B Inference); the workflow | ||
| posts an Actions notice and exits successfully. | ||
| - **Deterministic backbone**: the `textstat` delta always runs. The LLM judge is | ||
| supplementary and degrades gracefully when the W&B Inference key is absent. In | ||
| CI the key comes from the `WANDB_DOCS_INFERENCE_API_KEY` repo secret, passed to | ||
| the scorer as `WANDB_API_KEY`. | ||
| - **Localized content** under `ja/`, `ko/`, and `fr/` is skipped. | ||
|
|
||
| ## Run locally | ||
|
|
||
| ```bash | ||
| pip install -r scripts/readability/requirements.txt | ||
| git submodule update --init .claude | ||
| python3 scripts/readability/pr_report.py --base main --head HEAD | ||
| ``` | ||
|
|
||
| Add `--judge` to include the comprehension judge (requires `WANDB_API_KEY`). |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 9c13165 — added
continue-on-error: true. If the submodule fetch fails, the report step detects the missing analyzer and posts the informational fallback comment, so the check stays green and never blocks the PR.