feat: add opt-in test coverage analysis tool

[miteshashar]

Summary

• Adds node ./cli.js coverage task with 8 checks analyzing how thoroughly a schema's test files exercise its constraints
• Opt-in via coverage array in schema-validation.jsonc with object syntax supporting strict mode for CI enforcement
• Runs in CI (validate.yml); only strict: true schemas cause non-zero exit

Coverage Checks

Checks that enforce in CI (when strict: true)

1. Unused $defs — Flags $defs/definitions entries not referenced by any $ref. Dead definitions bloat schema size and confuse contributors who assume they're in use.

2. Description Coverage — Flags properties missing a description field. Descriptions power language server tooltips and editor autocomplete — without them, users get no guidance when editing config files.

3. Test Completeness — Checks that every top-level schema property appears in at least one positive test. Properties without test coverage can silently break across schema updates since no test validates them.

4. Enum Coverage — Checks that each enum value has positive test coverage and at least one invalid value in negative tests. Missing positive coverage means valid options are untested; missing negative coverage means the schema may silently accept typos.

5. Pattern Coverage — Checks that each pattern constraint has a matching positive test and a violating negative test. Untested patterns can contain regex bugs that either reject valid input or accept invalid input without anyone noticing.

6. Default Value Coverage — Checks that positive tests include non-default values for properties with default. Tests using only the default don't prove the property works with other values — a broken type constraint on a defaulted field would go undetected.

Heuristic checks (advisory only, never block CI)

7. Required Field Coverage — Checks that negative tests omit required fields, verifying the schema actually rejects incomplete input. Uses name-based matching (not path-aware), so returns warn to avoid false positives from repeated property names at different depths.

8. Negative Test Isolation — Flags negative test files that test multiple unrelated violation types simultaneously. Mixed-violation tests make it unclear which constraint the test is verifying, and a single fix can accidentally make the file pass for the wrong reason.

Demo Output

Running against claude-code-settings.json on current master (before any test improvements):

===== COVERAGE: claude-code-settings.json =====
✔️ 1. Unused $defs
  totalDefs: 3
❌ 2. Description Coverage
  totalProperties: 141
  missingCount: 19
  missing (19): statusLine.type, statusLine.command, statusLine.padding, fileSuggestion.type, ...
❌ 3. Test Completeness
  totalTopProperties: 49
  uncovered (13): allowManagedPermissionRulesOnly, awsAuthRefresh, awsCredentialExport, ...
❌ 4. Enum Coverage
  totalEnums: 7
  issues (10): autoUpdatesChannel — positive_uncovered — [latest], ...
❌ 5. Pattern Coverage
  totalPatterns: 3
  issues (4): allowedMcpServers[].serverName — no_negative_violation, ...
✔️ 6. Required Field Coverage
  totalRequiredGroups: 29
❌ 7. Default Value Coverage
  totalDefaults: 15
  issues (9): respectGitignore — only_default_tested — default=true, ...
✔️ 8. Negative Test Isolation
  totalNegativeTests: 9
===== 3 passed, 5 failed, 0 warned, 0 skipped =====

Test plan

• node ./cli.js check passes (existing checks unaffected)
• node ./cli.js coverage runs with empty coverage array (no-op)
• node ./cli.js coverage --schema-name=<schema>.json works for opted-in schemas
• strict: true causes exit(1) on failures; strict: false (default) exits 0
• CI runs coverage step without failing (empty array = no schemas to check)

🤖 Generated with Claude Code

Related

This tool's output was used to fix coverage for Claude Code Settings in #5385.

[madskristensen]

Super cool. @hyperupcall thoughts?