Skip to content

MST-9508: accept flow init auto-registration#637

Closed
tmatup wants to merge 1 commit intomainfrom
fix-MST-9508
Closed

MST-9508: accept flow init auto-registration#637
tmatup wants to merge 1 commit intomainfrom
fix-MST-9508

Conversation

@tmatup
Copy link
Copy Markdown
Member

@tmatup tmatup commented May 7, 2026

Summary

skill-flow-init-validate now grades the observable solution-first result instead of requiring one command spelling.

Current CLI behavior can register a flow project automatically when uip maestro flow init runs inside a solution. The captured run showed SolutionRegistration.Status=Registered and uip maestro flow validate WeatherAlert/WeatherAlert/WeatherAlert.flow returned Status=Valid, but the task failed because it required an explicit uip solution project add command.

Validation

  • Confirmed the remaining command criteria match the captured successful trace from run 2026-05-07_06-02-52.
  • Validated tests/tasks/uipath-maestro-flow/smoke/init_validate.yaml against TaskDefinition.
  • git diff --check

Jira: https://uipath.atlassian.net/browse/MST-9508

Root-cause from run 2026-05-07_06-02-52:\n\nMST-9508: skill-flow-init-validate failed only because the criterion required an explicit uip solution project add command, while uip maestro flow init inside a solution returned SolutionRegistration.Status=Registered and validate passed on WeatherAlert/WeatherAlert/WeatherAlert.flow. Removed the command-spelling assertion and kept the observable result checks.\n\nVerified: captured task.log command criteria match after the change; YAML validates with validate_yaml.py; git diff --check.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

Claude finished @tmatup's task in 1m 16s —— View job


Summary

Removes the command_executed success criterion for uip solution project add from the skill-flow-init-validate smoke task. The CLI auto-registers flow projects during uip maestro flow init when run inside a solution, making the explicit solution project add command unnecessary. The remaining criteria (solution creation, flow init, flow validate, --output json usage, and file existence) still cover the full workflow.

Change-by-Change Review

1. tests/tasks/uipath-maestro-flow/smoke/init_validate.yaml — removal of solution project add criterion

Severity: Low

The removed criterion (lines 65–72 in the old file) checked for uip solution project add. Since auto-registration handles this, removing it is correct — grading the observable outcome (file exists + validate passes) instead of prescribing one command spelling.

One consistency nit: The initial_prompt (lines 18–20) still says:

  1. Create the solution first.
  2. Create the Flow project inside that solution.
  3. Link the project to the solution.

Step 3 ("Link the project to the solution") now describes an action that is no longer validated and may be unnecessary if auto-registration fires. This won't break anything — an agent that runs the explicit command still passes, and one that doesn't also passes — but it could nudge agents toward a superfluous command. Consider removing or rewording step 3 (e.g., "Verify the project is registered in the solution") to match the updated grading. Fix this →

The remaining YAML structure is valid and all criteria weights/thresholds are consistent.

What's Missing

  • The initial_prompt step 3 is now slightly misleading (see above), but this is a minor polish item, not a blocker.
  • Nothing else identified.

Area Ratings

Area Status Notes
Frontmatter N/A No skill frontmatter changed
E2E Tests OK Task still has 5 success criteria covering the full workflow
Skill Body N/A No skill body changed
References & Assets N/A No references or assets changed
Repo Hygiene OK Change is scoped to the affected task file only

Issues for Manual Review

  • Confirm that uip maestro flow init reliably auto-registers in all environments (the PR cites one successful trace from 2026-05-07_06-02-52 — worth verifying this is consistent behavior, not a one-off).

Conclusion

Clean, well-scoped fix. The removal is justified by the CLI's auto-registration behavior. One optional polish item on initial_prompt step 3 wording. Approve.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

Claude finished @tmatup's task in 1m 22s —— View job


Coder-eval task lint (advisory)

1 task YAML changed; verdict counts: 0 Critical, 0 High, 1 Medium, 1 Low, 0 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

✅ Author confirms passing run: "Confirmed the remaining command criteria match the captured successful trace from run 2026-05-07_06-02-52."

Per-task lint

tests/tasks/uipath-maestro-flow/smoke/init_validate.yaml — verdict: Medium

Issues:

  • [Medium] Prompt over-specification (lines 17–20): initial_prompt prescribes a 3-step numbered procedure. Step 3 ("Link the project to the solution") is now vestigial — the uip solution project add criterion was removed by this PR, but the prompt still instructs the agent to do it. Two remaining steps still leak procedure rather than stating the goal.
  • [Low] Meaningful coverage (lines 32–69): All success_criteria are command_executed or file_exists. No criterion validates flow validate output (e.g., Status=Valid). A run where validate fails would still pass the test.

Suggested fixes:

  • Remove step 3 from the prompt (line 20) — it no longer corresponds to any criterion and could confuse the agent into running uip solution project add unnecessarily.
  • Consider condensing the remaining steps into a goal statement (e.g., "Create a solution, then create the Flow project inside it") to reduce over-specification.
  • Consider adding a run_command criterion that executes uip maestro flow validate WeatherAlert/WeatherAlert/WeatherAlert.flow --output json and checks expected_stdout for Valid or equivalent, so the test confirms validation actually passed.

Within-PR duplicates

  • No duplicate clusters detected (only 1 file changed).

Conclusion

⚠ 1 task has issues, max severity Medium. Advisory only — not blocking merge. The main concern is the vestigial step 3 in the prompt — since this PR specifically removes the corresponding criterion, the prompt should be updated to match.


@tmatup tmatup closed this May 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant