feat: add low-code agent evaluation docs

[mjnovice]

Summary

• Adds low-code agent evaluation documentation to the uipath-agents skill, filling a gap where coded agent evals have 5+ reference files but low-code has zero
• Full CLI support exists in uip agent eval (evaluator CRUD, eval set/test case management, run start/status/results/compare) but was undocumented in the skill
• 4 new reference files under references/lowcode/evaluation/ matching the structure of the coded eval docs
• Updates SKILL.md task navigation and lowcode.md capability registry with eval entries

Files Added

• evaluation/evaluate.md — entry point, prerequisites, file structure, key differences from coded evals
• evaluation/evaluators.md — 4 evaluator types (semantic-similarity, trajectory, context-precision, faithfulness), JSON format, custom prompts
• evaluation/evaluation-sets.md — eval set and test case CRUD, simulation options, JSON format
• evaluation/running-evaluations.md — run start/status/results/list/compare commands with output examples

Test plan

• Verify all internal links resolve (evaluate.md → evaluators.md, etc.)
• Verify SKILL.md task navigation row links to correct file
• Verify lowcode.md capability registry and "Read on demand" table entries
• Test documented commands against a real low-code agent project with uip agent eval

🤖 Generated with Claude Code

[github-actions]

Claude finished @mjnovice's task in 3m 0s —— View job

---

Summary

Adds 4 new reference files documenting the uip agent eval CLI for low-code agents under references/lowcode/evaluation/, plus task navigation and capability registry entries in SKILL.md and lowcode.md. Fills a real documentation gap — coded agent evals had extensive coverage while low-code had none.

Change-by-Change Review

1. skills/uipath-agents/SKILL.md (+1 line)

Severity: OK

New task navigation row correctly links to references/lowcode/evaluation/evaluate.md as the entry point and lists the three sub-docs in "Then". Placement is alphabetically consistent with surrounding rows. Pattern matches existing entries.

2. skills/uipath-agents/references/lowcode/lowcode.md (+2 lines)

Severity: OK

"Read on demand" table entry (line 49) and Capability Registry row (line 72) both link to evaluation/evaluate.md with correct relative paths. Sub-doc references in the "Then" column match the SKILL.md entry.

3. skills/uipath-agents/references/lowcode/evaluation/evaluate.md (new, 71 lines)

Severity: OK

Well-structured entry point. Quick Reference provides copy-paste commands, prerequisites are clear (push before run, local ops don't need auth), reference navigation links all resolve. Troubleshooting table covers common failure modes. Key Differences table vs coded evals is useful context.

Minor: Comments in the Quick Reference code block (lines 8, 11, 14 — # Add a test case, # Run evals and wait for results, # Check results...) restate what the adjacent command does. Per token-optimization rules, these could be stripped since the section heading "Quick Reference" already frames the context. Low — not blocking.

4. skills/uipath-agents/references/lowcode/evaluation/evaluators.md (new, 102 lines)

Severity: High — template variable inconsistency

Line 64 documents the trajectory evaluator's template variables as:

{{UserOrSyntheticInput}}, {{SimulationInstructions}}, {{ExpectedAgentBehavior}}, {{AgentRunHistory}}

But the Custom Prompts section at line 100 lists trajectory variables as:

{{AgentRunHistory}}, {{ExpectedBehavior}}

Two problems:

1. {{ExpectedAgentBehavior}} vs {{ExpectedBehavior}} — contradictory names. An agent following the Custom Prompts section would use the wrong variable.
2. Line 100 lists only 2 variables vs 4 on line 64 — the Custom Prompts section is incomplete, missing {{UserOrSyntheticInput}} and {{SimulationInstructions}}.

Fix: Reconcile the two lists — determine which variable name is correct ({{ExpectedAgentBehavior}} or {{ExpectedBehavior}}) by checking the actual CLI, then update the Custom Prompts section to list all 4 variables consistently. Fix this →

5. skills/uipath-agents/references/lowcode/evaluation/evaluation-sets.md (new, 140 lines)

Severity: OK

Comprehensive coverage of eval set and test case CRUD. Option tables are complete. Test case design section helpfully maps evaluator types to relevant flags. JSON format example is useful for agents that need to understand the file structure. Simulation options section is concise.

6. skills/uipath-agents/references/lowcode/evaluation/running-evaluations.md (new, 163 lines)

Severity: Low

Full coverage of run start/status/results/list/compare. JSON output examples match expected CLI patterns. Failure detection criteria are explicit — good for LLM usability.

The Workflow Example (lines 134–163) has numbered step comments (# 1. Push agent..., # 2. Add test cases..., etc.). These serve as section boundaries in a long code block, which is acceptable per token-optimization rules. Borderline — the surrounding ## Workflow Example heading provides no prose context, so the comments carry necessary structure here.

What's Missing

• E2E test tasks for low-code evaluation — No test task in tests/tasks/uipath-agents/ covers the new evaluation workflow (e.g., adding a test case, running an eval). Existing tasks cover guardrails, tools, contexts, escalations, etc., but not evals. Severity: Medium — consistent with the review guidelines that flag missing test coverage for new capabilities.
• Error handling guidance — The docs cover the happy path well but don't specify what an agent should do when uip agent eval run start returns a non-zero exit code or unexpected JSON structure. The troubleshooting table in evaluate.md partially covers this, but the sub-docs lack explicit "if this fails, do X" instructions. Severity: Low — the troubleshooting table provides reasonable coverage.

Area Ratings

Area	Status	Notes
Frontmatter	OK	No frontmatter changes needed — existing SKILL.md frontmatter unmodified
E2E Tests	Issue	No eval-specific test tasks added (Medium)
Skill Body	OK	Task navigation and capability registry entries correct
References & Assets	Issue	Template variable inconsistency in evaluators.md (High)
Repo Hygiene	OK	CODEOWNERS covers /skills/uipath-agents/, no secrets, no cross-skill refs, all links resolve

Issues for Manual Review

• CLI command accuracy — Cannot verify whether the documented uip agent eval subcommands, flags, and JSON output shapes match the actual CLI implementation. The commands follow established patterns from the coded eval docs, but the flag names (--simulate-input, --simulate-tools, --input-generation-instructions, --solution-id), JSON output codes (AgentEvalRunStarted, AgentEvalRunStatus, AgentEvalRunComparison), and evaluator type numeric mappings (5, 7, 8, 9) should be validated against the real CLI.
• Template variable names — Which is correct: {{ExpectedAgentBehavior}} or {{ExpectedBehavior}}? Only the CLI source or actual evaluator default prompts can resolve this.

Conclusion

Solid documentation addition that fills a clear gap. One High issue must be fixed before merge: the contradictory trajectory evaluator template variable names in evaluators.md (line 64 vs line 100). The missing e2e test task is Medium — consistent with the general test coverage gap across the repo. Everything else is clean.

---

[andreibalas-uipath]

The correct semantics for "pushing" an agent to Studio Web is actually to use the solutions CLI: uip solution upload. The command either creates a new solution or updates existing after solution is edited locally.