test(hitl): migrate HITL smoke tasks to canonical pattern vocabulary

[tmatup]

Summary

Replaces 12 brittle json_check.contains substring assertions across the five HITL smoke tasks with strict canonical-name checks aligned with references/hitl-patterns.md.

Task	Before	After
smoke_01_explicit	pattern contains pprov | rite | alid (3 fragments, OR @ 0.33)	pattern regex ^(approval-gate|write-back-validation)$
smoke_02_approval_gate	pattern contains pprov	pattern equals approval-gate
smoke_03_escalation	pattern contains scalat, insertion_point contains lassif	pattern regex ^(exception-escalation|agentic-output-review)$, insertion_point regex (?i)(classif|confidence|routing|...)
smoke_04_writeback	pattern contains rite | alid | nrich (3 fragments, OR @ 0.33)	pattern regex ^(write-back-validation|data-enrichment|agentic-output-review)$
smoke_05_compliance	pattern contains udit | omplianc | ign | pprov (4 fragments, OR @ 0.25)	pattern regex ^(compliance-checkpoint|approval-gate)$ (+ prompt strengthened with explicit "regulatory compliance" framing and GDPR Article 17 reference)

Each task's prompt now lists the same 6 canonical pattern machine names mirroring the section titles in references/hitl-patterns.md:

• approval-gate, exception-escalation, data-enrichment, compliance-checkpoint, write-back-validation, agentic-output-review

The agent is required to emit exactly one of those names verbatim. Where the scenario maps to multiple defensible canonical answers per the skill doc, the criterion uses regex and accepts each documented option.

Why

The original assertions used 4–6-character substrings of the canonical pattern names (e.g. expected: "scalat" for "escalation", expected: "udit" for "audit", expected: "ign" for "sign-off"). When the agent's free-form English answer didn't happen to contain that exact substring fragment, the smoke task failed — even when the answer was semantically correct. Local triage of run 2026-05-04_04-05-27 (and a 9-rep cross-grid across PR #517 / #545 / #546 worktrees) traced one such failure to expected: "scalat": smoke_03_escalation passed 8/9 locally but the failing rep emitted pattern: "confidence-gated human review" — a perfectly correct semantic answer that just didn't contain the literal substring.

Replacing the substring trap with a controlled vocabulary:

• Decouples the test from agent-paraphrase variance.
• Forces the agent to consult references/hitl-patterns.md (the prompt names the file) and pick a documented pattern.
• Surfaces test/skill drift loudly: if the skill ever renames a pattern, the test fails immediately rather than slowly drifting into uselessness.

Commits

1. fix(hitl): rewrite smoke_03_escalation around canonical pattern vocabulary — the originally failing test that triggered the investigation.
2. test(hitl): migrate 4 HITL smoke tasks to canonical pattern vocabulary — the proactive cleanup of the same anti-pattern in the four sibling smoke tasks.

Test plan

• Local: coder-eval run --repeats 3 against this branch — 3/3 SUCCESS at score 1.0 for smoke_03 (commit 1) and 12/12 SUCCESS at score 1.0 across smoke_01/02/04/05 (commit 2). 15/15 total.
• Authoritative validator-driven scan of origin/main (post-sync) confirms all 6 failing criteria across these 5 files are covered. After this PR lands: 0 violators repo-wide.
• CI smoke-skills run (will fire on PR push).

Companion PR

• UiPath/coder_eval#212 — adds the matching Pydantic validator that hard-fails any task YAML using json_check.contains with a < 8-char literal at task-load time. Merge this PR first, otherwise smoke-skills CI will go red the moment coder_eval ships the validator.

🤖 Generated with Claude Code

[github-actions]

Claude finished @tmatup's task in 2m 18s —— View job

---

PR Review: fix(hitl): rewrite smoke_03_escalation around canonical pattern vocabulary

• Read review criteria and project conventions
• Read PR diff and full changed file
• Read referenced skill docs (hitl-patterns.md) for context
• Check sibling smoke tests for the old anti-pattern
• Perform review and post feedback

---

Summary

Rewrites smoke_03_escalation.yaml to constrain the agent to a closed vocabulary of HITL pattern machine names instead of matching a fragile 5-character substring ("scalat"). The prompt now enumerates 6 canonical names and the assertion uses regex with an exact match against the two valid answers. This directly addresses stochastic CI failures.

Change-by-Change Review

1. tests/tasks/uipath-human-in-the-loop/smoke_03_escalation.yaml — prompt rewrite

Severity: OK (with two minor notes)

What's good:

• Switching from operator: contains / expected: "scalat" to operator: regex / expected: "^(exception-escalation|agentic-output-review)$" eliminates the failure mode where a semantically correct paraphrase misses a substring fragment. This is the right fix.
• Accepting both exception-escalation and agentic-output-review is well-reasoned — both patterns legitimately apply to a low-confidence-AI scenario per the skill doc, and the description explains why.
• The insertion_point regex ((?i)(classif|confidence|routing|low.{0,10}confidence|after.{0,30}\\bAI\\b)) is a solid improvement over "lassif" — it covers the semantic space without being overly brittle.
• The scenario text now uses the skill's own signal phrases ("edge cases", "cannot resolve autonomously", "escalate to a supervisor"), making it a better test of whether the agent recognises documented patterns.

Note 1 — pattern count mismatch (Low):
Line 37 says "enumerates six canonical patterns", but references/hitl-patterns.md actually has 10 ###-level sections (6 core + 4 domain-specific: IT/change-management, HR/offer, customer-communication, financial-transaction). The prompt explicitly lists the 6 names so the agent is still properly constrained — but an agent that reads the file will see 10, not 6. Consider saying "six core patterns" or updating the count.

Note 2 — compliance-checkpoint naming (Low):
Five of the six machine names are exact kebab-case conversions of hitl-patterns.md headings. The exception is compliance-checkpoint which truncates "Compliance and audit checkpoint". The prompt claims names "mirror the section titles" (line 36–37). This doesn't affect this test (neither name is in the accepted regex), but when the follow-up PR migrates the other smoke tests, smoke_05_compliance will presumably accept compliance-checkpoint — and the mismatch between that name and the heading could confuse the agent. Worth normalizing now or in that follow-up.

2. Task metadata (lines 1–6)

Severity: OK
task_id, description, and tags are unchanged and correct. tags follows convention: skill name first, then test type.

What's Missing

• Sibling smoke test migration (acknowledged): smoke_01, smoke_02, smoke_04, smoke_05 all still use the fragile operator: contains + substring pattern (confirmed via grep). The PR description explicitly calls this out as a follow-up — no issue, just confirming it's real.
• Nothing else identified.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill frontmatter changed
E2E Tests	OK	Smoke test improved; e2e tests untouched and still present
Skill Body	N/A	No skill doc changed
References & Assets	N/A	No reference files changed
Repo Hygiene	OK	Scoped to one file, no secrets, no cross-skill refs

Issues for Manual Review

• Verify that the coder_eval harness supports the regex operator on json_check assertions — the test assumes it does. If it only supports contains / equals, this will fail at runtime.
• The 3-rep local validation is promising but a small sample; CI run will be the real confirmation.

Conclusion

Clean, well-motivated fix. The core change (substring → canonical vocabulary + regex) is correct and directly addresses the CI flake. The two Low-severity notes (pattern count wording, compliance-checkpoint naming) are polish items that can land in this PR or the follow-up. No blocking issues.

---