docs(parity): expand harness to all 19 Dynamo parser families

[keivenchang]

Overview:

#9186 shipped the M2 cross-impl parity harness with 7 parser families; this PR adds the 12 remaining families Dynamo registers that have at least one upstream peer, bringing the matrix to full registry coverage.

Details:

• How is this fixed? New fixtures for pythonic, gemma4, deepseek_v3 (legacy), deepseek_v4, deepseek_v3_2, hermes, qwen25, mistral, jamba, llama3_json, phi4, and nemotron_nano; matching vLLM keys and SGLang detectors wired through tests/parity/parser/{vllm,sglang}.py; fixtures regenerated via the live Dynamo binding; cross-impl divergences recorded as KNOWN_DIVERGENCES xfails.
• Divergence categories surfaced: impl-defined recovery contracts on PARSER.batch.4/.5 (malformed args / missing end-token), trailing-text drops on PARSER.batch.8 (XML-style families), and format-detection failures on sglang/mistral/* and sglang/qwen25/* (root cause not investigated here; cells recorded as X so the matrix reflects observed behavior).
• No parser-side code change — lib/parsers/ untouched. Doc + harness wiring only.

Coverage matrix (as in tests/parity/README.md)

Cell legend: ✓ = both vLLM and SGLang match Dynamo; V = vLLM diverges (xfailed); S = SGLang diverges; VS = both diverge; n/a = no peer parser. Column legend: bN = PARSER.batch.N (1 happy-path, 2 multiple, 3 no-tool, 4 malformed, 5 missing-end-token, 6 empty-args, 7 complex-args, 8 interleaved-text, 9 empty, 10 duplicate).

family	b1	b2	b3	b4	b5	b6	b7	b8	b9	b10
Top-N models
deepseek_v4 § (DeepSeek V4)	✓	✓	✓	✓	V	✓	V	✓	✓	✓
gemma4 § (Gemma 4)	✓	✓	✓	✓	✓	✓	✓	V	✓	✓
glm47 (GLM 5.1)	✓	✓	✓	✓	VS	✓	✓	V	✓	✓
harmony † (gpt-oss)	S	S	✓	S	S	S	S	S	✓	S
kimi_k2 (Kimi K2.6)	✓	✓	✓	S	✓	✓	✓	VS	✓	✓
minimax_m2 (MiniMax 2.7)	✓	✓	✓	V	VS	✓	✓	VS	✓	✓
nemotron_deci †§ (Nemotron)	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
qwen3_coder (Qwen 3.5)	✓	✓	✓	✓	✓	✓	✓	VS	✓	✓
Others
deepseek_v3	✓	✓	✓	V	VS	✓	✓	S	✓	✓
deepseek_v3_1	✓	✓	✓	V	VS	✓	✓	S	✓	✓
deepseek_v3_2	✓	✓	✓	✓	V	✓	V	S	✓	✓
hermes	✓	✓	✓	VS	✓	✓	✓	V	✓	✓
jamba §	✓	✓	✓	✓	V	✓	✓	V	✓	✓
llama3_json §	✓	V	✓	V	V	✓	✓	✓	✓	V
mistral	S	S	✓	VS	S	S	S	VS	✓	S
nemotron_nano †§	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a	n/a
phi4 §	✓	✓	✓	✓	V	✓	V	V	✓	✓
pythonic	✓	✓	✓	✓	✓	✓	✓	VS	✓	✓
qwen25 †	S	S	✓	S	S	S	S	S	✓	S

† = no vLLM peer (or vLLM returns UNAVAILABLE at runtime — harmony needs token IDs); § = no SGLang peer.

Verification:

Smoke against pinned containers — vllm 120/30/40, sglang 83/37/70, dynamo 190/190 (passed/xfailed/skipped); skips are families without a peer on that side.

Where should the reviewer start?

tests/parity/README.md (matrix), then tests/parity/parser/test_parity_parser.py (KNOWN_DIVERGENCES block).

Related Issues:

Relates to DIS-1906

/coderabbit profile chill

[coderabbitai]

Walkthrough

This PR updates the parity test suite documentation to reflect expanded parser family coverage. The README was revised to display three additional families in the results table, increase aggregate cell counts from 140 to 200, and adjust narrative text to correctly reference the broader fixture corpus and its implications for test case overlap.

Changes

Parser Family Coverage Documentation Update

Layer / File(s)	Summary
Parity Status Table tests/parity/README.md	The "Current parity status (M2, batch mode)" results table extended to include pythonic, gemma4, and deepseek_v3 families; aggregate cell count increased from 140 to 200 excluding Dynamo oracle, with corresponding revisions to parity, xfail, and n/a tallies.
Documentation References tests/parity/README.md	Narrative text updated to state reviewers can grep case IDs across all 10 families; overlap metric against Rust suite revised from ~70 to ~100 cases to align with current fixture expansion.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately reflects the main change—expanding parity harness coverage to all 19 Dynamo parser families—matching the PR objectives.
Description check	✅ Passed	The PR description provides comprehensive details including overview, implementation specifics, coverage matrix, verification results, and related issues.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

tests/parity/README.md (2)

329-329: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Update the case count to match the expanded coverage.

Line 329 still references "~70 black-box tests", which was accurate for 7 families × 10 cases. With 10 families now covered, this should be ~100 to match line 380's updated reference.

📝 Proposed fix

-— for the ~70 black-box "given input X, parser returns Y" tests,
+— for the ~100 black-box "given input X, parser returns Y" tests,

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/parity/README.md` at line 329, Update the README text that still reads
"~70 black-box \"given input X, parser returns Y\" tests" to reflect the
expanded coverage; change that phrase to "~100 black-box \"given input X, parser
returns Y\" tests" (or "about 100") so it matches the updated reference on line
380 and the current 10 families × 10 cases count; locate the exact string in
tests/parity/README.md and replace it accordingly.

---

102-102: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Update the test count to reflect the expanded coverage.

The cost metric still references 210 tests, which was correct for 7 families (7 families × 3 implementations × 10 cases = 210). With the addition of pythonic, gemma4, and deepseek_v3, the total is now 10 families × 3 implementations × 10 cases = 300 tests.

📊 Proposed fix

-| **Cost** | (TBD) | ~3 s for 210 tests | ~60 s for 30 tests (server boot dominates) |
+| **Cost** | (TBD) | ~3 s for 300 tests | ~60 s for 30 tests (server boot dominates) |

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/parity/README.md` at line 102, The README table's Cost row still lists
"210 tests"; update the test count to reflect the new total of 300 tests (10
families × 3 implementations × 10 cases) by replacing "210 tests" with "300
tests" in the table cell (the line starting with "| **Cost** | (TBD) | ~3 s for
210 tests | ..."). Ensure the text exactly reflects "300 tests".

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@tests/parity/README.md`:
- Line 329: Update the README text that still reads "~70 black-box \"given input
X, parser returns Y\" tests" to reflect the expanded coverage; change that
phrase to "~100 black-box \"given input X, parser returns Y\" tests" (or "about
100") so it matches the updated reference on line 380 and the current 10
families × 10 cases count; locate the exact string in tests/parity/README.md and
replace it accordingly.
- Line 102: The README table's Cost row still lists "210 tests"; update the test
count to reflect the new total of 300 tests (10 families × 3 implementations ×
10 cases) by replacing "210 tests" with "300 tests" in the table cell (the line
starting with "| **Cost** | (TBD) | ~3 s for 210 tests | ..."). Ensure the text
exactly reflects "300 tests".

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f478b20b-d2f4-4675-9294-b97fade3e24e

📥 Commits

Reviewing files that changed from the base of the PR and between f5a9463 and a7d8090.

📒 Files selected for processing (1)

• tests/parity/README.md

[keivenchang]

@coderabbitai both outside-diff findings caught — fixed in 51ddee6, but with larger numbers than your suggestions because the matrix grew further in-PR after a7d8090:

• Line 102 cost row: 210 → 570 (19 families × 3 impls × 10 cases, not 300 — added hermes, qwen25, mistral, jamba, llama3_json, phi4, nemotron_nano, deepseek_v3_2 after your snapshot).
• Line 346 black-box test count: ~70 → ~190 (19 families × 10 cases, not ~100).

Same fix shape, scaled up. Please verify and resolve.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check current state of the README around the claimed fix locations
grep -n "570\|210\|300\|cost\|190\|~70\|~100\|families" tests/parity/README.md | head -60
echo "---LINE 95-115---"
sed -n '95,115p' tests/parity/README.md
echo "---LINE 340-355---"
sed -n '340,355p' tests/parity/README.md

Length of output: 2558

---

@keivenchang: Both fixes verified in the current branch ✅

• Line 102 → ~3 s for 570 tests (19 × 3 × 10 = 570) ✓
• Line 346 → ~190 black-box "given input X, parser returns Y" tests (19 × 10 = 190) ✓

One minor stale reference I noticed while scanning: Line 250 still reads "across all 10 families" while Line 144 now correctly states "19 parser families total". Not blocking, but worth a one-word fix if you do another pass.

Resolving the original comments now.

[resolve]

[furionw]

Thank you