ci: Readability-score delta Github Action#2884
Conversation
Depends on a skill in coreweave/docs-skills#42 Adding a submodule reference is a followup action after that PR merges, some time before merging this into wandb/docs The submodule will depend on the DOCENGINE_TOKEN already in wandb/docs for cross-org authentication.
Points the .claude submodule at coreweave/docs-skills main (11d7344), which lands the readability delta analyzer and AI comprehension scorer (#42) that the readability-delta CI check imports from .claude/scripts/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HiveMind Sessions1 session · 8m · $2.23
View all sessions in HiveMind → Run |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
There was a problem hiding this comment.
Pull request overview
Adds a new CI workflow that computes and posts an informational readability delta (word-weighted across changed English .mdx files) as an upserted PR comment, backed by a Python reporting script and unit tests. It optionally includes an AI-agent-comprehension judge via W&B Inference when the relevant secret is present.
Changes:
- Introduces
scripts/readability/pr_report.pyto diff base/head, score changed pages via the docs-skills submodule analyzer, aggregate results, and build Markdown. - Adds a new workflow
.github/workflows/readability-delta.ymlto run on PRs touching.mdxand post/upsert a single PR comment. - Adds supporting docs, requirements, and pytest coverage for parsing/aggregation/Markdown generation.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/readability/pr_report.py |
Implements the diff → score → aggregate → Markdown report pipeline, plus optional LLM judge integration. |
scripts/readability/README.md |
Documents the readability-delta check, dependencies, behavior, and local run instructions. |
scripts/readability/requirements.txt |
Declares Python dependencies for deterministic scoring and optional judge. |
scripts/readability/tests/test_pr_report.py |
Unit tests for parsing changed files, aggregation, and Markdown formatting; includes an optional integration test. |
scripts/readability/tests/conftest.py |
Registers the integration pytest marker to avoid unknown-mark warnings. |
scripts/readability/tests/__init__.py |
Marks the tests directory as a package for test discovery/import behavior. |
.github/workflows/readability-delta.yml |
New workflow to run the script, write a fallback notice on failure, and upsert a single PR comment. |
.github/workflows/README.md |
Adds a section describing the new readability-delta workflow and its configuration/auth. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| status = parts[0] | ||
| # Renames/copies: use the new path (last field). | ||
| path = parts[-1] | ||
| if not path.endswith(".mdx"): | ||
| continue | ||
| if path.startswith(_LOCALE_PREFIXES): | ||
| continue | ||
| entries.append({"status": status[0], "path": path}) |
There was a problem hiding this comment.
Good catch — fixed in 9c13165. parse_changed_files now keeps old_path for rename/copy rows (old_path = parts[1]), so scoring can read the base version from the old path.
| path = entry["path"] | ||
| status = entry["status"] | ||
|
|
||
| if status == "D": | ||
| return {"path": path, "status": "deleted"} | ||
|
|
||
| before_text = "" if status == "A" else git_show(f"{base}:{path}", repo_root) | ||
| after_text = git_show(f"{head}:{path}", repo_root) |
There was a problem hiding this comment.
Fixed in 9c13165. score_entry now reads the before-text from old_path, and the result row carries old_path so the comprehension-judge loop does the same. A renamed + edited page now scores as a real delta instead of being reported as new.
| # Rename rows have three tab fields: status, old path, new path. | ||
| name_status = "R096\tmodels/old.mdx\tmodels/renamed.mdx" | ||
| entries = pr_report.parse_changed_files(name_status) | ||
| assert entries == [{"status": "R", "path": "models/renamed.mdx"}] |
There was a problem hiding this comment.
Updated in 9c13165 — the rename test now asserts old_path is retained. I also added test_score_entry_rename_reads_base_from_old_path (mocks git_show) so a regression that reads base from the new path fails the suite.
| - name: Check out docs-skills submodule | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| env: | ||
| DOCENGINE_TOKEN: ${{ secrets.DOCENGINE_TOKEN }} |
There was a problem hiding this comment.
Fixed in 9c13165 — added continue-on-error: true. If the submodule fetch fails, the report step detects the missing analyzer and posts the informational fallback comment, so the check stays green and never blocks the PR.
| - name: Install dependencies | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| run: pip install -r scripts/readability/requirements.txt |
There was a problem hiding this comment.
Fixed in 9c13165 — added continue-on-error: true. If dependency install fails, the report step's try/fallback writes the informational notice instead of failing the job.
| textstat | ||
| # AI agent comprehension judge (optional; only when WANDB_API_KEY is set). | ||
| # Imported lazily from the submodule's _docs_eval_lib.py: | ||
| openai | ||
| weave |
There was a problem hiding this comment.
Fixed in 9c13165 — pinned textstat==0.7.13, openai==2.45.0, weave==0.53.1 (the versions verified working in CI and locally), matching the repo's == pinning convention.
📚 Mintlify Preview Links✨ Added (1 total)⚙️ Other (1)
📝 Changed (2 total)📄 Pages (1)
⚙️ Other (1)
🤖 Generated automatically when Mintlify deployment succeeds |
…degradation - Preserve the old path for renamed/copied .mdx so the base version is read from the old path; a renamed+edited page is now scored as a delta instead of being misreported as brand-new. Covers both the deterministic scorer and the comprehension judge. Adds parse + score_entry unit tests. - Pin actions/checkout, actions/setup-python, actions/cache to the commit SHAs already used across this repo's workflows (supply-chain hardening). - Mark the submodule-fetch and dependency-install steps continue-on-error so a transient failure degrades to the informational fallback comment instead of failing the check, matching the workflow's "never blocks a PR" contract. - Pin textstat/openai/weave to the verified-working versions, matching the repo's requirements-pinning convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These are noise: We are using these tools to evaluate readability but we are not shipping these packages anywhere. |
|
Thanks for the review, @copilot — all nine comments had merit and are addressed in 9c13165: Rename scoring (correctness) — Action SHA pinning — "Never blocks a PR" contract — the submodule-fetch and dependency-install steps are now Dependency pinning — All 14 unit tests pass and the workflow YAML validates. Ready for another look. |
| f"**{summary['weighted_fk_delta']:+.1f} ({summary['direction']})**." | ||
| ) | ||
| else: | ||
| lines.append("No pages had enough prose to score a readability delta.") |
There was a problem hiding this comment.
Agreed — fixed in dffc509. aggregate() returns None for any PR with no scorable before-and-after page (add-only, delete-only, or insufficient prose), so the message now reads "No changed page had a scorable before-and-after version, so there is no readability delta to report. See the per-page details below." and defers specifics to the per-page table.
aggregate() returns None whenever no page has a scorable before-and-after version — which includes PRs that only add new pages or only delete pages, not just insufficient prose. Generalize the message and point at the per-page table for specifics. Addresses Copilot review follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Temporary prose simplification (FK 10.0 -> 9.5, "easier") to trigger and demonstrate the readability-delta workflow on this PR. REVERT before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🔗 Link Checker Results✅ All links are valid! No broken links were detected. Checked against: https://wb-21fd5541-docs-2626.mintlify.app |
Readability impactWord-weighted Flesch-Kincaid grade change across 1 changed page: -0.5 (easier). Lower Flesch-Kincaid grade and higher reading ease both mean easier to read. This check is informational and never blocks a PR. Human readability
AI agent comprehensionRated 0-3 (higher is easier for an agent to parse and act on).
Curated-docs baseline median FK grade by type: conceptual 10.5, procedural 8.8, reference 9.4. |
| weighted_delta = round(weighted_sum / weight_total, 1) | ||
| if weighted_delta < 0: | ||
| direction = "easier" | ||
| elif weighted_delta > 0: | ||
| direction = "harder" | ||
| else: | ||
| direction = "unchanged" |
| - name: Install dependencies | ||
| # Non-fatal: if deps fail to install, the report step still runs and | ||
| # falls back to the informational notice rather than failing the PR. | ||
| continue-on-error: true | ||
| if: steps.fork.outputs.is_fork != 'true' | ||
| run: pip install -r scripts/readability/requirements.txt |
| --- | ||
|
|
||
| Files are uploaded to a Google Cloud bucket managed by W&B when you log artifacts. The contents of the bucket are encrypted both at rest and in transit. Artifact files are only visible to users who have access to the corresponding project. | ||
| When you log artifacts, W&B uploads the files to a Google Cloud bucket. W&B encrypts the bucket at rest and in transit. Only users with access to the project can see the files. |
| run: | | ||
| git config --global url."https://x-access-token:${DOCENGINE_TOKEN}@github.com/".insteadOf "https://github.com/" | ||
| git submodule update --init --recursive .claude |
Description
Posts an informational, non-blocking PR comment describing how the PR affects the readability of the English docs it changes. It reports the delta (before/after) for well-established formulas (Flesch-Kincaid grade, Flesch reading ease, Gunning fog, SMOG), word-weighted across the changed pages, plus an optional AI-agent-comprehension rating from a W&B Inference LLM judge.
Details
job a no-op (still reporting success). Forks are uncommon in
wandb/docsand coreweave repos cannot use forks at all.DOCENGINE_TOKENalready inwandb/docsfor cross-org authentication.WANDB_DOCS_INFERENCE_API_KEYsecret (a W&B API key whose entity has Inference credits), passed to the scorer asWANDB_API_KEY. When that secret is absent, the deterministictextstatdelta still runs.Fixes DOCS-2626
Testing
mint dev)mint broken-links)