Skip to content

add qa-flow, consolidate QA/AQA/testgen skills, add manual tests#110

Draft
sveto wants to merge 46 commits into
mainfrom
qa-aqa-testgen-skills
Draft

add qa-flow, consolidate QA/AQA/testgen skills, add manual tests#110
sveto wants to merge 46 commits into
mainfrom
qa-aqa-testgen-skills

Conversation

@sveto

@sveto sveto commented Jun 11, 2026

Copy link
Copy Markdown
Contributor
  • QA, AQA, Testgen workflows tested and bugfixed/hardenen in several places.

  • QA and AQA are now structurally unified.

  • Some common parts of the 3 workflows, along with the skills created previously by Maksym, are merged into 4 pre-existing skills and 2 new skills foreseen by docs/definitions/skills.md.

  • Two unforeseen skills are created and added to the canonical list: qa-knowledge and qa-structure.

  • Instructions on how to test the workflows are also added: docs/manual-tests.

  • The idea of 'canonical lists' is now explicitly mentioned in coding-agents-prompt-authoring skill (previously it was only mentioned in a reference file).

@sveto sveto marked this pull request as ready for review June 11, 2026 08:40
This was referenced Jun 11, 2026
@github-actions github-actions Bot added the enhancement New feature or request label Jun 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Rosetta Triage Review

Summary: This PR consolidates and significantly refactors the QA, AQA, and Testgen AI agent workflows — introducing a brand-new end-to-end qa-flow (10 phase files), two new skills (discovery, scenarios-generation), and hardening all three workflow families across both r2 and r3 instruction trees. It also improves the validate-prompts CI pipeline and adds manual test guides for all three flows.

Note — Instruction-quality review: This PR changes instructions/r*/**, so automated instruction-quality analysis was performed in addition to standard code review.


Findings:

[CRITICAL] operation-manager/SKILL.md — Agent-agnostic violation
Frontmatter allowed-tools: Bash(npx:*) is Claude Code-specific; npx rosettify@latest is Node.js-specific. Per Rosetta prompt hardening rules, skills must be coding-agent-agnostic. The fallback path (ACQUIRE todo-tasks-fallback.md FROM KB) exists but is not clearly surfaced for non-Claude-Code agents. The model: claude-sonnet-4-6 frontmatter field is also Claude Code-specific.
Recommendation: Document explicitly that this skill requires Node.js + Claude Code, or restructure so the fallback is the default path with the npx variant as an opt-in.

[HIGH] Multiple files — Frontmatter description exceeds 30-token cap

  • discovery/SKILL.md: ~46 tokens. Suggested: "Gather source artifacts from systems-of-record (Jira/Confluence/TestRail) into a phase-defined raw-context artifact. Read-only."
  • scenarios-generation/SKILL.md: ~36 tokens. Remove the inline contrast-with-testing clause (it belongs in <when_to_use_skill>).
  • qa-flow.md: ~97 tokens, hardcodes vendor names. Suggested: "Backend API test automation end-to-end: from test cases to automated tests, with HITL gates at specification and correction phases."
  • orchestrator-contract/SKILL.md: ~40 tokens. Suggested: "MUST activate when acting as orchestrator: defines delegation, dispatch, routing, and review protocol."

[HIGH] discovery/SKILL.md and orchestrator-contract/SKILL.md — Missing <when_to_use_skill> section
Both skills lack the schema-required <when_to_use_skill> section that disambiguates when an agent should load this skill vs a peer. All other new skills in this PR include it.

[HIGH] operation-manager/SKILL.md — SRP violation (dual orchestrator/subagent role)
The <process> section defines two complete independent flows (Orchestrator flow + Subagent flow). Recommendation: add a clear self-selection gate at the top or extract one flow to a reference.

[HIGH] qa-flow-project-config-loading.md — Step numbering collision + template bulk
Steps 0.2a and 0.2 appear in that order, which is contradictory. Fix: renumber sequentially. Also, 113 of 276 lines are template content — templates belong in references/ per hardening rules. Recommendation: extract to references/qa-project-config-loading-templates.md.

[MEDIUM] DRY — Duplicated validation rules in discovery/SKILL.md and scenarios-generation/SKILL.md
Both skills restate the sensitive-data redaction rule in both <core_concepts> and <validation_checklist>. The checklist entry should be a pointer to <core_concepts>, not a restatement.

[MEDIUM] orchestrator-contract/SKILL.md — Inline subagent dispatch template should be in references/
The boilerplate 42-line dispatch template embedded in <process> is reference material. Extracting it would reduce the file from 140 to ~90 lines.

[MEDIUM] qa-flow.md — HITL skip-rules partially duplicate hitl skill
<skip_rules> contains gate-execution logic that the hitl skill owns canonically. Keep only the qa-flow-specific precondition table; defer gate execution to USE SKILL hitl.

[POSITIVE] CI improvement in validate-prompts.yml
Significantly improved pipeline: structured JSON output, severity-based gate (blocks on HIGH+), PR comment summary with link to full per-file details in Actions run summary.

[POSITIVE] Manual test docs
docs/manual-tests/ is an excellent addition — auth-free testing modes (Mode A/B), per-phase checklists, and "try to break it" sections make the flows much easier to validate manually.


Suggestions:

  • Consider adding a docs/definitions/skills.md reference to the Rosetta onboarding docs — this PR surfaces the issue that the canonical skills list is not discoverable to contributors.
  • For operation-manager: the inline canonical next output JSON example belongs in references/rosettify-next-output.md with a pointer from the skill body.
  • PR title could be more descriptive — e.g. feat: add qa-flow, consolidate QA/AQA/testgen skills, add manual tests.

Automated triage by Rosetta agent

@sveto sveto changed the title Qa aqa testgen skills add qa-flow, consolidate QA/AQA/testgen skills, add manual tests Jun 11, 2026
sveto added 9 commits June 11, 2026 12:39
# Conflicts:
#	instructions/r2/core/skills/debugging/SKILL.md
#	instructions/r2/core/skills/orchestrator-contract/SKILL.md
#	instructions/r2/core/skills/requirements-authoring/SKILL.md
#	instructions/r2/core/workflows/aqa-flow-data-collection.md
#	instructions/r2/core/workflows/aqa-flow-requirements-clarification.md
#	instructions/r2/core/workflows/aqa-flow-selector-identification.md
#	instructions/r2/core/workflows/aqa-flow-test-correction.md
#	instructions/r2/core/workflows/aqa-flow-test-implementation.md
#	instructions/r2/core/workflows/aqa-flow-test-report-analysis.md
#	instructions/r2/core/workflows/aqa-flow.md
#	instructions/r2/core/workflows/testgen-flow-data-collection.md
#	instructions/r2/core/workflows/testgen-flow-gap-and-contradiction-analysis.md
#	instructions/r2/core/workflows/testgen-flow-project-config-loading.md
#	instructions/r2/core/workflows/testgen-flow-question-generation.md
#	instructions/r2/core/workflows/testgen-flow-requirements-document-generation.md
#	instructions/r2/core/workflows/testgen-flow-test-case-export.md
#	instructions/r2/core/workflows/testgen-flow-test-case-generation.md
#	instructions/r2/core/workflows/testgen-flow.md
#	instructions/r3/core/skills/debugging/SKILL.md
#	instructions/r3/core/skills/orchestrator-contract/SKILL.md
#	instructions/r3/core/skills/requirements-authoring/SKILL.md
#	instructions/r3/core/workflows/aqa-flow-data-collection.md
#	instructions/r3/core/workflows/aqa-flow-requirements-clarification.md
#	instructions/r3/core/workflows/aqa-flow-selector-identification.md
#	instructions/r3/core/workflows/aqa-flow-test-correction.md
#	instructions/r3/core/workflows/aqa-flow-test-implementation.md
#	instructions/r3/core/workflows/aqa-flow-test-report-analysis.md
#	instructions/r3/core/workflows/aqa-flow.md
#	instructions/r3/core/workflows/testgen-flow-data-collection.md
#	instructions/r3/core/workflows/testgen-flow-gap-and-contradiction-analysis.md
#	instructions/r3/core/workflows/testgen-flow-project-config-loading.md
#	instructions/r3/core/workflows/testgen-flow-question-generation.md
#	instructions/r3/core/workflows/testgen-flow-requirements-document-generation.md
#	instructions/r3/core/workflows/testgen-flow-test-case-export.md
#	instructions/r3/core/workflows/testgen-flow-test-case-generation.md
#	instructions/r3/core/workflows/testgen-flow.md
#	plugins/core-claude/hooks/hooks.json
#	plugins/core-claude/skills/debugging/SKILL.md
#	plugins/core-claude/skills/orchestrator-contract/SKILL.md
#	plugins/core-claude/skills/requirements-authoring/SKILL.md
#	plugins/core-claude/workflows/INDEX.md
#	plugins/core-claude/workflows/aqa-flow-data-collection.md
#	plugins/core-claude/workflows/aqa-flow-requirements-clarification.md
#	plugins/core-claude/workflows/aqa-flow-selector-identification.md
#	plugins/core-claude/workflows/aqa-flow-test-correction.md
#	plugins/core-claude/workflows/aqa-flow-test-implementation.md
#	plugins/core-claude/workflows/aqa-flow-test-report-analysis.md
#	plugins/core-claude/workflows/aqa-flow.md
#	plugins/core-claude/workflows/testgen-flow-data-collection.md
#	plugins/core-claude/workflows/testgen-flow-gap-and-contradiction-analysis.md
#	plugins/core-claude/workflows/testgen-flow-project-config-loading.md
#	plugins/core-claude/workflows/testgen-flow-question-generation.md
#	plugins/core-claude/workflows/testgen-flow-requirements-document-generation.md
#	plugins/core-claude/workflows/testgen-flow-test-case-export.md
#	plugins/core-claude/workflows/testgen-flow-test-case-generation.md
#	plugins/core-claude/workflows/testgen-flow.md
#	plugins/core-codex/.agents/skills/debugging/SKILL.md
#	plugins/core-codex/.agents/skills/orchestrator-contract/SKILL.md
#	plugins/core-codex/.agents/skills/requirements-authoring/SKILL.md
#	plugins/core-codex/.agents/workflows/INDEX.md
#	plugins/core-codex/.agents/workflows/aqa-flow-data-collection.md
#	plugins/core-codex/.agents/workflows/aqa-flow-requirements-clarification.md
#	plugins/core-codex/.agents/workflows/aqa-flow-selector-identification.md
#	plugins/core-codex/.agents/workflows/aqa-flow-test-correction.md
#	plugins/core-codex/.agents/workflows/aqa-flow-test-implementation.md
#	plugins/core-codex/.agents/workflows/aqa-flow-test-report-analysis.md
#	plugins/core-codex/.agents/workflows/aqa-flow.md
#	plugins/core-codex/.agents/workflows/testgen-flow-data-collection.md
#	plugins/core-codex/.agents/workflows/testgen-flow-gap-and-contradiction-analysis.md
#	plugins/core-codex/.agents/workflows/testgen-flow-project-config-loading.md
#	plugins/core-codex/.agents/workflows/testgen-flow-question-generation.md
#	plugins/core-codex/.agents/workflows/testgen-flow-requirements-document-generation.md
#	plugins/core-codex/.agents/workflows/testgen-flow-test-case-export.md
#	plugins/core-codex/.agents/workflows/testgen-flow-test-case-generation.md
#	plugins/core-codex/.agents/workflows/testgen-flow.md
#	plugins/core-codex/.codex-plugin/hooks.json
#	plugins/core-codex/.codex/hooks.json
#	plugins/core-copilot-standalone/.github/instructions/plugin-files-mode.instructions.md
#	plugins/core-copilot-standalone/.github/prompts/INDEX.md
#	plugins/core-copilot-standalone/.github/prompts/aqa-flow-data-collection.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/aqa-flow-requirements-clarification.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/aqa-flow-selector-identification.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/aqa-flow-test-correction.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/aqa-flow-test-implementation.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/aqa-flow-test-report-analysis.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/aqa-flow.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/testgen-flow-data-collection.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/testgen-flow-gap-and-contradiction-analysis.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/testgen-flow-project-config-loading.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/testgen-flow-question-generation.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/testgen-flow-requirements-document-generation.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/testgen-flow-test-case-export.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/testgen-flow-test-case-generation.prompt.md
#	plugins/core-copilot-standalone/.github/prompts/testgen-flow.prompt.md
#	plugins/core-copilot-standalone/.github/skills/debugging/SKILL.md
#	plugins/core-copilot-standalone/.github/skills/orchestrator-contract/SKILL.md
#	plugins/core-copilot-standalone/.github/skills/requirements-authoring/SKILL.md
#	plugins/core-copilot/.github/plugin/hooks.json
#	plugins/core-copilot/commands/INDEX.md
#	plugins/core-copilot/commands/aqa-flow-data-collection.md
#	plugins/core-copilot/commands/aqa-flow-requirements-clarification.md
#	plugins/core-copilot/commands/aqa-flow-selector-identification.md
#	plugins/core-copilot/commands/aqa-flow-test-correction.md
#	plugins/core-copilot/commands/aqa-flow-test-implementation.md
#	plugins/core-copilot/commands/aqa-flow-test-report-analysis.md
#	plugins/core-copilot/commands/aqa-flow.md
#	plugins/core-copilot/commands/testgen-flow-data-collection.md
#	plugins/core-copilot/commands/testgen-flow-gap-and-contradiction-analysis.md
#	plugins/core-copilot/commands/testgen-flow-project-config-loading.md
#	plugins/core-copilot/commands/testgen-flow-question-generation.md
#	plugins/core-copilot/commands/testgen-flow-requirements-document-generation.md
#	plugins/core-copilot/commands/testgen-flow-test-case-export.md
#	plugins/core-copilot/commands/testgen-flow-test-case-generation.md
#	plugins/core-copilot/commands/testgen-flow.md
#	plugins/core-copilot/hooks.json
#	plugins/core-copilot/skills/debugging/SKILL.md
#	plugins/core-copilot/skills/orchestrator-contract/SKILL.md
#	plugins/core-copilot/skills/requirements-authoring/SKILL.md
#	plugins/core-cursor-standalone/.cursor/commands/INDEX.md
#	plugins/core-cursor-standalone/.cursor/commands/aqa-flow-data-collection.md
#	plugins/core-cursor-standalone/.cursor/commands/aqa-flow-requirements-clarification.md
#	plugins/core-cursor-standalone/.cursor/commands/aqa-flow-selector-identification.md
#	plugins/core-cursor-standalone/.cursor/commands/aqa-flow-test-correction.md
#	plugins/core-cursor-standalone/.cursor/commands/aqa-flow-test-implementation.md
#	plugins/core-cursor-standalone/.cursor/commands/aqa-flow-test-report-analysis.md
#	plugins/core-cursor-standalone/.cursor/commands/aqa-flow.md
#	plugins/core-cursor-standalone/.cursor/commands/testgen-flow-data-collection.md
#	plugins/core-cursor-standalone/.cursor/commands/testgen-flow-gap-and-contradiction-analysis.md
#	plugins/core-cursor-standalone/.cursor/commands/testgen-flow-project-config-loading.md
#	plugins/core-cursor-standalone/.cursor/commands/testgen-flow-question-generation.md
#	plugins/core-cursor-standalone/.cursor/commands/testgen-flow-requirements-document-generation.md
#	plugins/core-cursor-standalone/.cursor/commands/testgen-flow-test-case-export.md
#	plugins/core-cursor-standalone/.cursor/commands/testgen-flow-test-case-generation.md
#	plugins/core-cursor-standalone/.cursor/commands/testgen-flow.md
#	plugins/core-cursor-standalone/.cursor/rules/plugin-files-mode.mdc
#	plugins/core-cursor-standalone/.cursor/skills/debugging/SKILL.md
#	plugins/core-cursor-standalone/.cursor/skills/orchestrator-contract/SKILL.md
#	plugins/core-cursor-standalone/.cursor/skills/requirements-authoring/SKILL.md
#	plugins/core-cursor/commands/INDEX.md
#	plugins/core-cursor/commands/aqa-flow-data-collection.md
#	plugins/core-cursor/commands/aqa-flow-requirements-clarification.md
#	plugins/core-cursor/commands/aqa-flow-selector-identification.md
#	plugins/core-cursor/commands/aqa-flow-test-correction.md
#	plugins/core-cursor/commands/aqa-flow-test-implementation.md
#	plugins/core-cursor/commands/aqa-flow-test-report-analysis.md
#	plugins/core-cursor/commands/aqa-flow.md
#	plugins/core-cursor/commands/testgen-flow-data-collection.md
#	plugins/core-cursor/commands/testgen-flow-gap-and-contradiction-analysis.md
#	plugins/core-cursor/commands/testgen-flow-project-config-loading.md
#	plugins/core-cursor/commands/testgen-flow-question-generation.md
#	plugins/core-cursor/commands/testgen-flow-requirements-document-generation.md
#	plugins/core-cursor/commands/testgen-flow-test-case-export.md
#	plugins/core-cursor/commands/testgen-flow-test-case-generation.md
#	plugins/core-cursor/commands/testgen-flow.md
#	plugins/core-cursor/skills/debugging/SKILL.md
#	plugins/core-cursor/skills/orchestrator-contract/SKILL.md
#	plugins/core-cursor/skills/requirements-authoring/SKILL.md
Comment thread instructions/r3/core/workflows/qa-flow.md Outdated
@mkuznietsov

Copy link
Copy Markdown
Contributor

'qa-flow' if it's API flow for user it may not be obvious that for API testing we need to trigger qa-flow and for UI we need to trigger aqa-flow.

I think we need to change names to proper one. Please rename flows.

@mkuznietsov

Copy link
Copy Markdown
Contributor

Why r2 instructions changed? Do we need fix on prod?

@mkuznietsov mkuznietsov self-requested a review June 15, 2026 15:57

@mkuznietsov mkuznietsov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix review comments

@sveto

sveto commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

Why r2 instructions changed? Do we need fix on prod?

I'm not quite sure -- and my initial plan was not to touch r2 at all, but such configuration does not pass pre-commit hooks.

Comment thread instructions/r3/core/skills/discovery/references/confluence-binding.md Outdated
Present-and-wait gate before an irreversible / high-stakes step (apply corrections, approve a spec/plan). The calling phase supplies: the **closed approval-token list**, the **re-present step**, and the **revisit target** on full reject. Approval vocabulary is governed by `hitl` — this is its QA-flow specialization; the phase's closed token list is authoritative for that phase.

1. Present the artifact for review; **WAIT** for explicit approval. Do NOT assume approval; a message containing questions or suggestions is reviewing, not approving.
2. **Approval = an exact closed token** (case-insensitive — `APPROVED` / `Approve` / `yes` all match the lowercase token). Anything else — `"looks good"`, `"ship it"`, `"LGTM"`, `"sounds good"`, `"go ahead"`, `"OK"`, `"go"`, a question, a suggestion, or silence — is **REVIEW, not approval**: re-prompt for an exact token. The list is **closed**; "or similar" / "etc." extension language in other loaded rules does NOT extend it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m thinking these instructions could be part of the HITL skill. Why were they defined in a separate file?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Approval vocabulary is governed by hitl — this is its QA-flow specialization". It gets its own parameters, new for each phase; hitl does not accept parameters.

But what is true is that 1-2 were repeating hitl in many ways. I've tried to fix it.


</resources>

</requirements-authoring>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This skill now is more than 500 rows. I assume that it's not recommended size. Maybe it's worth to make it smaller?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, shortening the file is not exactly the scope of this PR -- but the time has come. Please take a look at the result.

Comment thread instructions/r3/core/skills/scenarios-generation/SKILL.md Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Rosetta Instruction Quality Review — Description Hardening

Confirming @YevheniiaLementova's finding. The frontmatter description on scenarios-generation/SKILL.md line 3 violates the pattern defined in pa-hardening.md line 106:

Frontmatter's description must be call-to-action and extremely small and dense (<30 tokens!)

Current description:

Rosetta — design test scenarios, cases, and specs from requirements: GWT API specs, TMS-format cases, config-resolved exports.

Concrete issues:

  • Rosetta — prefix is redundant — every SKILL.md in this directory is a Rosetta skill; no other skill uses this prefix
  • Not call-to-action — the standard pattern starts with To followed by an imperative verb (see requirements-authoring, coding-agents-prompt-authoring, debugging, etc.)
  • Enumerates all three modes verbatim (GWT API specs, TMS-format cases, config-resolved exports) — that level of detail belongs in <when_to_use_skill>, not the frontmatter description

Comparison with existing skills:

Skill Description
requirements-authoring To author, update, and validate functional/non-functional requirements as atomic units with user approval.
coding-agents-prompt-authoring To author, adapt, review, and validate prompts (skills, agents, workflows, rules, etc.) with brief, contracts, and a validation pack.
debugging To investigate errors, test failures, and unexpected behavior — root cause before fix.

Suggested fix (~13 tokens, call-to-action):

To design test scenarios, GWT specs, and TMS cases from requirements or API contracts.

Automated triage by Rosetta agent

Comment thread instructions/r3/core/skills/discovery/references/confluence-binding.md Outdated
Comment thread instructions/r3/core/skills/discovery/SKILL.md Outdated
mkuznietsov
mkuznietsov previously approved these changes Jun 26, 2026

@mkuznietsov mkuznietsov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@isolomatov-gd isolomatov-gd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls check comments, understand the failure mode and FIX that failure across the board. NOT only files provided

Comment thread instructions/r3/core/skills/debugging/SKILL.md Outdated
Comment thread instructions/r3/core/skills/debugging/SKILL.md Outdated
Comment thread instructions/r3/core/skills/debugging/SKILL.md Outdated
Comment thread instructions/r3/core/skills/debugging/SKILL.md Outdated

</implementation>

<test_execution_triage>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is tightly coupled with the workflow.
Workflow KNOWS how to and calls the skill, SKILL does NOT know workflow, SKILL is universal functionality, that workflow wants to reuse.

Comment thread instructions/r3/core/skills/qa-knowledge/assets/api-analysis-template.md Outdated
Comment thread instructions/r3/core/skills/qa-knowledge/SKILL.md Outdated
Comment thread instructions/r3/core/skills/qa-knowledge/SKILL.md Outdated
Comment thread instructions/r3/core/skills/qa-knowledge/SKILL.md Outdated
Comment thread instructions/r3/core/skills/qa-structure/assets/api-qa-project-config-template.md Outdated
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

📋 Prompt Quality Validation Report

❌ Validation Failed

The full markdown report and raw JSON output are available in the workflow artifacts for 5 days.


Files With Issues

  • instructions/r3/core/skills/coding-agents-prompt-authoring/SKILL.md: 1 issue(s)
  • instructions/r3/core/skills/data-collection/references/documentation-vendor-binding.md: 1 issue(s)
  • instructions/r3/core/skills/data-collection/references/tms-vendor-binding.md: 1 issue(s)
  • instructions/r3/core/skills/testing/SKILL.md: 1 issue(s)
  • instructions/r3/core/skills/qa-structure/references/config-schema.md: 1 issue(s)
  • instructions/r3/core/skills/scenarios-generation/SKILL.md: 1 issue(s)
  • instructions/r3/core/workflows/api-qa-flow.md: 1 issue(s)
  • instructions/r3/core/workflows/api-qa-flow-api-spec-analysis.md: 1 issue(s)
  • instructions/r3/core/workflows/api-qa-flow-test-case-specification.md: 1 issue(s)
  • instructions/r3/core/workflows/api-qa-flow-execution-and-report-analysis.md: 1 issue(s)
  • instructions/r3/core/workflows/testgen-flow.md: 1 issue(s)
  • instructions/r3/core/workflows/testgen-flow-data-collection.md: 1 issue(s)
  • instructions/r3/core/workflows/testgen-flow-project-config-loading.md: 2 issue(s)
  • instructions/r3/core/workflows/testgen-flow-gap-and-contradiction-analysis.md: 2 issue(s)
  • instructions/r3/core/workflows/testgen-flow-question-generation.md: 1 issue(s)
  • instructions/r3/core/workflows/testgen-flow-requirements-document-generation.md: 1 issue(s)
  • instructions/r3/core/workflows/testgen-flow-test-case-generation.md: 1 issue(s)
  • instructions/r3/core/workflows/ui-qa-flow.md: 1 issue(s)
  • instructions/r3/core/workflows/ui-qa-flow-selector-implementation.md: 1 issue(s)

📄 instructions/r3/core/skills/coding-agents-prompt-authoring/SKILL.md

⚠️ Issues Found

Severity Gate Details
Medium Rosetta Problem:
The new <rosetta_canonical_lists> block (lines 134-142) inlines Rosetta-specific canonical-list references (docs/definitions/workflows.md, templates.md, agents.md, skills.md, rules.md) directly into the always-loaded general SKILL.md. This content previously lived only in pa-rosetta.md, the on-demand reference that is explicitly scoped to Rosetta prompts. The skill is general and coding-agent-agnostic, and its own core_concepts state that specifics are kept in references/* to keep this file small.
Reason:
Every agent loading this general skill, including for non-Rosetta targets, now carries target-specific detail inline. This is a mild progressive-disclosure and bloat regression against the skill's own stated principle of pushing specifics into references.
Solution:
Keep the canonical-lists pointer in pa-rosetta.md (the Rosetta-only, on-demand reference) instead of inlining it in SKILL.md; if a visibility cue is desired, a single line such as 'for Rosetta targets see pa-rosetta.md' is enough.

📄 instructions/r3/core/skills/data-collection/references/documentation-vendor-binding.md

⚠️ Issues Found

Severity Gate Details
High Rosetta Problem:
The 'Redaction targets' section deep-links into another skill's private reference: qa-knowledge/references/redaction-scope.md. The parent SKILL.md already names sensitive-data as the 'canonical authority' for redaction. This same deep-link is repeated verbatim in all three vendor bindings.
Reason:
Cross-skill deep-linking breaks skill-folder isolation: if qa-knowledge relocates or renames redaction-scope.md, three data-collection bindings silently break. It also creates dual redaction authority (sensitive-data vs qa-knowledge's scope file), which can confuse which one governs.
Solution:
Reference the redaction authority only through the sensitive-data skill (as SKILL.md does), and let that skill own its scope internally; drop the direct path into qa-knowledge's private references. If a shared scope file is truly needed, expose it as a non-private/shared asset rather than reaching into a sibling skill's folder.

📄 instructions/r3/core/skills/data-collection/references/tms-vendor-binding.md

⚠️ Issues Found

Severity Gate Details
Medium Rosetta Problem:
The 'Redaction targets' section names a sibling skill and its internals as rationale: step/precondition/custom-field data 're-emit downstream ... via the scenarios-generation TestRail export binding back into the shared TestRail project.' This is sibling-awareness plus non-operational explanatory provenance about another skill's internal binding.
Reason:
Referencing a sibling skill's internal binding couples this file to scenarios-generation's implementation and adds non-operational rationale the hardening guidance says to strip; if that skill's design changes, the explanation goes stale while adding no behavioral value.
Solution:
State the operational reason redaction matters (these fields re-emit into version-controlled downstream artifacts) without naming the scenarios-generation skill or its internal 'export binding'; keep the binding unaware of which downstream skill consumes its output.

📄 instructions/r3/core/skills/testing/SKILL.md

⚠️ Issues Found

Severity Gate Details
High Conflict Resolution Problem:
The new <implementation_modes> add API and UI impl modes that author integration/E2E tests making real HTTP/browser calls (the reference shows API tests hitting http://localhost:8080 and asserting response time), but the unchanged core_concepts rule 'MUST enforce 1-second timeout on EACH test' and the <validation_checklist> item 'Each test completes within 1-second timeout' still apply as a blanket bar with no carve-out for these modes.
Reason:
An agent running API/UI impl mode and then the mandatory validation checklist would flag its own valid E2E tests as failures against the 1s bar, or stall on the contradiction.
Solution:
Scope the 1-second timeout bar to unit tests only, or add an explicit exception in the new impl-modes validation line stating API/UI/E2E tests use a mode-appropriate timeout.

📄 instructions/r3/core/skills/qa-structure/references/config-schema.md

⚠️ Issues Found

Severity Gate Details
Medium Rosetta Problem:
The Consumed by column pins keys to exact consumer step numbers inside sibling phase files (e.g. api-qa-flow-data-collection.md (step 1.2b), api-qa-flow-api-spec-analysis.md step 2.1, step 3 cross-check). A skill reference reaching into a phase's internal step numbering is tighter cross-layer coupling than pa-rosetta boundaries encourage, and the step ids go stale if a phase renumbers its steps.
Reason:
Loose coupling keeps the SSoT schema valid when consuming phases are edited; step-level references silently rot and mislead a maintainer verifying where a key is used.
Solution:
Keep the phase-file name as the consumer cue but drop the internal step numbers (e.g. api-qa-flow-data-collection.md instead of ... (step 1.2b)), so the schema references the consumer contract without depending on a phase's private step layout.

📄 instructions/r3/core/skills/scenarios-generation/SKILL.md

⚠️ Issues Found

Severity Gate Details
High Rosetta Problem:
The new SKILL.md frontmatter (lines 1-7) has name, description, license, tags, baseSchema but omits disable-model-invocation and user-invocable. The skill schema (docs/schemas/skill.md) marks both as REQUIRED ("always set explicitly, even when equal to the default"), and every sibling r3 skill (testing, debugging, load-context) declares both.
Reason:
Missing required invocation/discovery fields make the skill's auto-invocation and /-menu behavior non-deterministic across agents and inconsistent with the schema contract every other skill honors.
Solution:
Add disable-model-invocation: false and user-invocable: true (or the intended values) to the frontmatter block.

📄 instructions/r3/core/workflows/api-qa-flow.md

⚠️ Issues Found

Severity Gate Details
Medium Bloat Control Problem:
<description_and_purpose> (line 13) contains an author-facing meta/provenance note: "(Source-system + tool enumeration owned by the frontmatter description field — not restated here.)". This narrates authoring intent to the reader rather than instructing the executing agent.
Reason:
Provenance/rationale notes inside a target prompt add tokens without changing agent behavior and drift toward the injected-rationale AI-slop pattern the authoring skill warns against.
Solution:
Drop the parenthetical; the frontmatter already carries the enumeration, so the executing agent needs no note explaining where it lives.

📄 instructions/r3/core/workflows/api-qa-flow-api-spec-analysis.md

⚠️ Issues Found

Severity Gate Details
Low Failure Handling Problem:
Step 2.2 (line 59) ACQUIREs qa-knowledge/assets/api-analysis-template.md and calls it "load-bearing", but only the redaction-scope ACQUIRE is fail-closed (<redaction_contract>, line 81). A zero-document ACQUIRE for the api-analysis template has no defined stop/ask behavior, unlike sibling phases (e.g. project-config-loading fail-closes its template ACQUIRE and the parent workflow has a zero-doc rule).
Reason:
Without it, a KB miss on the output-shape asset leaves the agent to fabricate the artifact structure from memory, producing an off-contract api-analysis.md that downstream phases treat as authoritative.
Solution:
Add a one-line fail-closed clause for the api-analysis-template.md ACQUIRE (zero docs → stop and report, do not emit from memory), mirroring the redaction-scope fail-closed and the parent <failure_handling> zero-doc rule.

📄 instructions/r3/core/workflows/api-qa-flow-test-case-specification.md

⚠️ Issues Found

Severity Gate Details
Low Bloat Control Problem:
Step 4.2 item 1 is one dense multi-clause run-on that stacks the ACQUIRE order, the output-contract ownership rationale, the scenario-taxonomy ownership, and the scenarios-generation skill invocation into a single sentence, against the schema guidance for phase files ('imperative bullet points, shorter lines').
Reason:
A phase file consumed step-by-step is easier to execute reliably when each atomic action is its own bullet; a packed sentence raises the chance the agent skips a sub-clause (e.g. the ACQUIRE-first ordering).
Solution:
Split step 4.2.1 into separate bullets: (a) ACQUIRE the template, (b) USE SKILL scenarios-generation with the step-4.1 inputs, (c) the one-line contract-ownership note. Do not remove content, only break the sentence apart.

📄 instructions/r3/core/workflows/api-qa-flow-execution-and-report-analysis.md

⚠️ Issues Found

Severity Gate Details
Medium Structural Coherence Problem:
<phase_steps> enumerates four steps (1 obtain results, 2 run triage, 3 review findings, 4 update state) but the executable step blocks collapse steps 1 and 2 into a single 6.1 block, so the four listed steps do not map 1:1 to numbered step blocks (unlike the sibling spec phase where five phase_steps map cleanly to 4.1-4.5).
Reason:
An agent that plans its todo list from <phase_steps> will look for a distinct 'obtain results' step and find it folded inside 6.1, creating minor tracking ambiguity about whether that step ran.
Solution:
Either split obtain-results into its own step="6.1" block and renumber triage to 6.2, or reword <phase_steps> to three items so the outline matches the 6.1/6.2/6.3 blocks.

📄 instructions/r3/core/workflows/testgen-flow.md

⚠️ Issues Found

Severity Gate Details
Medium Bloat Control Problem:
The <orchestration_and_escalation> and gate-type-convention blocks carry non-operational explanatory meta-commentary, including an explicit comparison to sibling workflows: it differs from ui-qa-flow/api-qa-flow, where every pause is a type=HITL/HITL-CONDITIONAL gate. This is sibling-workflow awareness and rationale text rather than an executable rule.
Reason:
Reduces cognitive load in the densest part of the workflow and avoids cross-family/sibling awareness that a workflow prompt should not depend on.
Solution:
Drop the ui-qa-flow/api-qa-flow comparison sentence and trim the parenthetical rationale; keep only the operative statement that Phases 3 and 6 carry type=HITL and the priority-(3) pauses do not.

📄 instructions/r3/core/workflows/testgen-flow-data-collection.md

⚠️ Issues Found

Severity Gate Details
Medium Rosetta Problem:
The phase repeatedly names the internal reference files that the data-collection skill loads, e.g. data-collection loads references/issue-vendor-binding.md and data-collection loads references/documentation-vendor-binding.md. A caller of a skill should not describe the skill's private internals.
Reason:
Keeps the skill boundary intact so that internal refactors of data-collection do not silently invalidate this phase file. (Net Rosetta compliance still much better than base, which hardcoded mcp_Jira_MCP_* names.)
Solution:
State only that the phase invokes data-collection with the resolved vendor binding and passes the ticket key / Confluence handles; remove the references to the skill's internal references/*-vendor-binding.md files.

📄 instructions/r3/core/workflows/testgen-flow-project-config-loading.md

⚠️ Issues Found

Severity Gate Details
Medium Rosetta Problem:
The redaction gate reaches directly into another skill's private references: MUST ACQUIRE qa-knowledge/references/redaction-scope.md FROM KB. This is cross-skill deep-linking into qa-knowledge's internal references/ content rather than consuming it through a skill entry point.
Reason:
Deep-linking couples this phase to qa-knowledge's internal file layout; a rename/move of that reference breaks the mandatory fail-closed redaction gate. (The redaction gate itself is a strong addition over base.)
Solution:
Prefer routing the redaction re-scan through the sensitive-data skill (already referenced) or a documented shared-asset path, rather than ACQUIRE-ing another skill's private reference file directly.
Medium Precision & Explicitness Problem:
The config file moved from a shared, reusable location (base: found/created in the repo's agent-specific directory, and explicitly used for any future queries related to this project) to a per-ticket copy at plans/testgen-{TICKET-KEY}/testgen-project-config.md (one copy per run, not a shared project-wide file). The base's cross-ticket reuse capability is dropped, so the config-elicitation question is re-asked on every ticket for the same project.
Reason:
Base's stated goal of reusing the config for future queries on the same project is silently lost; teams processing many tickets on one project must re-answer the retrieval/auth setup each time.
Solution:
If cross-ticket reuse is still intended, add a step to look for and reuse an existing project-level config before creating the per-ticket copy (or state explicitly why reuse was intentionally removed).

📄 instructions/r3/core/workflows/testgen-flow-gap-and-contradiction-analysis.md

⚠️ Issues Found

Severity Gate Details
High Workflow Completeness Problem:
The zero-issues branch (update_state step 2.4) marks Phase 3 SKIPPED and jumps to Phase 4, so questions.md and answers.md are never created. The parent testgen-flow.md self-check / final validation expects those phase artifacts to exist, so a clean zero-findings run leaves the workflow's artifact-existence validation unable to confirm completion.
Reason:
A workflow whose own final validation cannot confirm completion on a valid (zero-findings) run is unreliable and may loop, stall, or falsely report failure.
Solution:
On the zero-issues path, still produce the expected Phase 3 artifacts (e.g. an empty/notes questions.md and answers.md marked 'no questions'), or update the parent's artifact-existence validation to treat these as optional when Phase 2 yields zero findings.
High Conflict Resolution Problem:
The <update_state> zero-issues branch instructs: Mark Phase 3 as SKIPPED — no issues from Phase 2 ... then proceed to Phase 4. This directly contradicts the parent testgen-flow.md, whose <workflow_phases> and <orchestration_and_escalation> priority-(2) rule state the Phase 3 / Phase 6 HITL gates are never skipped by user instruction and never overridden. The rewrite introduces a cross-file contradiction that did not exist in base (base had no Phase-3-skip path).
Reason:
An agent following the parent workflow will refuse the skip the phase file tells it to perform (or vice versa), producing inconsistent behavior at the most safety-relevant HITL gate.
Solution:
Reconcile the two files: either carve an explicit zero-issues exception into the parent's Phase 3 gate rule (e.g. gate is satisfied vacuously when Phase 2 yields zero findings), or have Phase 2 still enter Phase 3, which then closes immediately with no questions, instead of marking Phase 3 SKIPPED.

📄 instructions/r3/core/workflows/testgen-flow-question-generation.md

⚠️ Issues Found

Severity Gate Details
High Safety Boundaries Problem:
Phase 3 (modified) writes user-typed answers.md into the tracked plans/testgen-{TICKET-KEY}/ folder with no redaction pass, even though users answer config/auth questions there. Phase 0 explicitly guards against exactly this credential-leak path, and the parallel api-qa and other new phases run a fail-closed redaction gate before writing.
Reason:
User answers routinely contain credentials, tokens, and environment details; writing them unredacted to a version-controlled file leaks secrets — the highest-severity failure class this PR otherwise defends against.
Solution:
Add a redaction step before writing answers.md: route through the sensitive-data skill (or the same redaction gate the other phases use) so secrets/PII in user answers are masked before the file is committed.

📄 instructions/r3/core/workflows/testgen-flow-requirements-document-generation.md

⚠️ Issues Found

Severity Gate Details
Medium Bloat Control Problem:
The <failure_handling> section adds a multi-paragraph 'Conscious tradeoff — why no inline per-entry fallback (declared once, not re-derived per turn)' block that justifies the design choice (three bullets on skill-as-hard-dependency, deployment guarantee, phase-owned contract, plus 'This tradeoff is intentional and bounded to this phase'). The operational instruction it supports ('the phase blocks when the skill is unavailable; do NOT fabricate a partial requirements.md') is already stated in the same section's 'Skill execution failure' bullet.
Reason:
Design-justification prose is analyst-artifact content projected into a target prompt; it re-sends every turn and dilutes the executable instructions without changing agent behavior.
Solution:
Keep the single operational rule (block + do-not-fabricate) and delete the design-rationale prose that explains WHY the fallback was omitted; move that reasoning to the change-log/blueprint if retention is desired.

📄 instructions/r3/core/workflows/testgen-flow-test-case-generation.md

⚠️ Issues Found

Severity Gate Details
Medium Safety Boundaries Problem:
Phase 5 (modified) writes test-scenarios.md into the tracked plans/testgen-{TICKET-KEY}/ folder with no redaction pass, while the parallel api-qa spec phase redacts before writing. Test data is a named highest-risk field for embedded credentials/PII.
Reason:
Generated test scenarios can embed real sample credentials or PII; an unredacted write to a tracked file risks leaking them into version control.
Solution:
Add the same fail-closed redaction gate (via sensitive-data) before emitting test-scenarios.md, consistent with the api-qa spec phase and Phase 0's stated redaction discipline.

📄 instructions/r3/core/workflows/ui-qa-flow.md

⚠️ Issues Found

Severity Gate Details
Low Input Contract Problem:
The Phase 1 (data_collection) header states Input as 'user request + CONTEXT.md + ARCHITECTURE.md + IMPLEMENTATION.md', but the actual Phase 1 file ui-qa-flow-data-collection.md declares its real inputs as the TestRail case ID/URL and Confluence page ID/search terms. The parent's per-phase Input line does not match the phase's own input contract.
Reason:
An orchestrator or reader trusting the parent's per-phase Input line may prepare or expect the wrong inputs at the Phase 1 handoff, adding confusion versus the phase file's true contract.
Solution:
Align the parent Phase 1 Input line to the external-source inputs the phase actually consumes (TestRail case ID/URL, Confluence page ID/search terms), or mark the repo docs as optional background context rather than the phase inputs.

📄 instructions/r3/core/workflows/ui-qa-flow-selector-implementation.md

⚠️ Issues Found

Severity Gate Details
Medium Failure Handling Problem:
This phase has no dedicated failure_handling block. Its part_a_inventory_gate tells the agent to record a blocked status in agents/ui-qa-state.md and ask the user, but it assumes the state file exists and the <test-name> slug resolves. The sibling phases (test-implementation, test-report-analysis, test-correction) all carry an explicit 'state file missing or slug unresolvable' branch; this phase omits it.
Reason:
Without this branch, a run that reaches Phase 5 with a lost/renamed state file has no defined behavior, so the agent may fabricate a slug or silently create a new state file, diverging from the other phases and losing the audit trail.
Solution:
Add a short failure branch mirroring the siblings: if agents/ui-qa-state.md is missing or the <test-name> slug is unresolvable, stop, report in chat, ask the user to restore the state file, and do not auto-recreate it or guess the slug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants