add qa-flow, consolidate QA/AQA/testgen skills, add manual tests

docs/definitions/skills.md

@@ -28,6 +28,8 @@
 - coding
 - testing
 - debugging
+- qa-structure
+- qa-knowledge
 - codemap
 - load-context
 - load-project-context

docs/definitions/workflows.md

@@ -4,7 +4,9 @@
 - research-flow
 - code-analysis-flow
 - external-lib-flow
-- aqa-flow
+- ui-qa-flow
+- api-qa-flow
+- testgen-flow
 - modernization-flow
 - adhoc-flow
 - coding-agents-prompting-flow

docs/manual-tests/api-qa-flow.md

@@ -0,0 +1,83 @@
+# API-QA flow -- manual test guide
+
+End-to-end smoke check for the backend API test-automation workflow
+(`api-qa-flow.md`, 8 phases 0–7). External data is pulled by the single `discovery` skill
+via config-resolved **vendor bindings** (`jira` / `testrail` / `confluence`). Two shared
+skills carry the cross-phase scaffolding -- **`qa-structure`** (canonical
+paths, `{IDENTIFIER}`, state-file shape) and **`qa-knowledge`** (failure taxonomies,
+redaction scope, and the per-phase artifact skeletons, `ACQUIRE`'d as assets at point of
+use). UI-QA uses the same two.
+
+## Prerequisites
+
+- [ ] Rosetta plugin installed and active -- exercise the workflow via the installed plugin (plugin mode), not the raw r3 instructions
+- [ ] Working dir lets you write under `agents/api-qa/`
+- [ ] Sample backend repo at `RefSrc/<project>/` **or** a Swagger URL handy
+- [ ] **Auth is optional** -- see *Auth-free / mock testing* below. For a full real run: Atlassian MCP (Jira/Confluence) and/or TestRail MCP authenticated.
+
+## Auth-free / mock testing
+
+You do **not** need real Jira/Confluence/TestRail credentials to test this flow. The `discovery` bindings only ever make *real* MCP calls or stop with a gap -- they never fabricate. Two ways to run auth-free:
+
+- **Mode A: source out-of-scope / provided (no MCP call).** In `agents/api-qa/api-qa-project-config.md` leave the vendor's in-scope keys unset (`testrail_base_url`, `jira_base_url`, `confluence_base_url` → `N/A`). Phase 1 then resolves **`SKIPPED_NO_CONFIG`**, records the gap, and proceeds on what you supply directly (paste case fields / use the direct-description trigger, or a Swagger URL/`RefSrc` path for the API contract). Validates degradation + the whole downstream pipeline on your canned input. *Caveat:* the bindings have no "provided-inline retrieval" path, so if you **do** set an in-scope key, Phase 1 will attempt the real MCP and stop on auth failure.
+- **Mode B: stub MCP (canned data).** Point the Atlassian/TestRail MCP at a local stub that answers the tool calls (`jira_get_issue`, `confluence_*`, `mcp_testrail_*`) with fixture JSON. The binding treats the fixtures like a real MCP, running the full extract → normalize → redact → write path with zero real auth. Guardrails permit this (*"User can override (mocked data)"*).
+
+> What Mode A actually validates: the **degradation** path (gap/skip, no fabrication), not the real data-pull. Use Mode B to exercise the pull logic without credentials.
+
+## Trigger prompt (pick one)
+
+```
+Write backend API tests for TC-1234.
+Swagger: https://api.example.com/swagger.json
+```
+```
+Automate backend tests for PROJ-123 with Swagger from RefSrc/my-backend/docs/openapi.json
+```
+```
+Create API tests for the user registration endpoint (no ticket, direct description).
+```
+```
+Generate API tests for POST /orders and GET /orders/{id} from RefSrc/orders-svc/. TestRail suite: S-42.
+```
+```
+Write contract tests for the auth endpoints (login / refresh / logout). Swagger: https://api.example.com/v2/openapi.yaml; Jira: PROJ-789.
+```
+
+## Per-phase quick checks
+
+| Phase | HITL | File to inspect | Skill(s) | Must see |
+|---|---|---|---|---|
+| 0 - Config Loading | conditional | `agents/api-qa/api-qa-project-config.md` + `agents/api-qa/{IDENTIFIER}/initial-data.md` | `qa-structure`, `questioning` (only if config missing) | Config carries the required keys (each a real value **or** `N/A -- <reason>`): `documentation_mcp_collection_skill`, `confluence_base_url`, `swagger_url`, `spec_format`, `backend_source_path`, `system`, `testrail_base_url`, `jira_base_url`, `testcase_mcp_collection_skill`, `project_id`/`suite_id`, `framework`, `mechanism`. If config didn't pre-exist you were asked the data-retrieval intake question. |
+| 1 - Data Collection | -- | `agents/api-qa/{IDENTIFIER}/raw-data.md` | `discovery` (`testrail`/`jira` + `confluence` documentation MCP, inline in step 1.2b), `reverse-engineering` | Sections: Test Case Data · Documentation (or `SKIPPED_NO_CONFIG` outcome) · Existing Test Patterns · Backend Source Code Analysis · API Endpoints Identified · Summary. **No literal `.env` values / passwords.** |
+| 2 - API Spec Analysis | -- | `agents/api-qa/{IDENTIFIER}/api-analysis.md` | `reverse-engineering` (API-contract mode), `sensitive-data` | Every target endpoint has a contract entry OR is flagged a gap with reason; `Source: hybrid` entries have a non-empty Notes/Discrepancies field (reconciliation or explicit `None.`). |
+| 3 - Gap & Requirements Clarification | **HITL** | `agents/api-qa/{IDENTIFIER}/analysis.md` | `requirements-use` (gap_analysis mode), `questioning` | All 7 sections (Gaps `G[N]` · Contradictions `C[N]` · Ambiguities `A[N]` · Questions Critical/Important/Optional · Answers · Resolutions · Open Assumptions). Workflow **paused** with concrete questions; after answers, the invariant `Questions == Answers + Open Assumptions + Skipped + Deferred` holds and no Critical sits in a BLOCKING ASSUMPTION state. |
+| 4 - Test Case Specification | **HITL** | `agents/api-qa/{IDENTIFIER}/test-specs.md` | `scenarios-generation` (gwt_spec mode), `sensitive-data` | `ATC-NNN` Given-When-Then entries each trace to a Phase 3 requirement; Summary + Test File Mapping + Shared Utilities + Execution Order present. Workflow **paused** for approval -- exact token `approved`/`approve`/`yes`. |
+| 5 - Test Implementation | **HITL** | Test files at project layout + hand-off summary returned inline | `testing` (API impl mode), `coding` (repo conventions) | Every ATC has a test fn with its `ATC-NNN` in name/docstring **or** is surfaced under `### Gaps`; lint/format clean; **no hardcoded credentials**; workflow **paused** for you to execute tests (the phase does not run them). |
+| 6 - Execution & Report Analysis | **HITL** | `agents/api-qa/{IDENTIFIER}/execution-report.md` | `debugging` (triage mode), `sensitive-data` | Execution Summary + Failures-by-Category (8-item API taxonomy: Connection/Environment · Authentication · Request · Response Assertion · Test Data · Timing/Race · Application Bug · Unknown) + per-failure Evidence label; **no literal tokens / Authorization headers**; workflow **paused** for test-report input. |
+| 7 - Test Corrections | **HITL** | Proposed Changes block | `debugging`, `coding` | Per-change proposal (Source root cause · File · In-scope · Change type · Before/After · Reason · Impact · Risk · Approval). After approval, **only test files modified -- NOT application source**. Iteration cap **3 cycles per change**, then escalate. |
+
+State file: `agents/api-qa-state.md` (`## Phase Completion Status` + per-phase append blocks).
+
+**Across all phases:** `qa-structure` supplies the canonical paths, `{IDENTIFIER}`, and state-file shape; `qa-knowledge` supplies the failure taxonomy, the artifact skeletons (each `ACQUIRE`'d FROM KB at the step that writes it -- expect to see those loads), and the redaction scope. The Skill(s) column lists each phase's domain skills on top of these two. The Phase 0 config interview, config template, and config schema are `qa-structure` assets/references; the tiny QA state-stub + initial-data skeletons stay inline in the phase.
+
+## Try to break it
+
+| Action | Expected behavior |
+|---|---|
+| Run with no test-case reference at all (no ticket, case ID, or description) | **Phase 0** stops first: `Phase 0 blocked: test case reference unresolvable from initial prompt`; asks for a TestRail ID / Jira key / feature name -- no fabricated `{IDENTIFIER}` (Phase 0 guarantee) |
+| Give a feature description (yields a valid `{IDENTIFIER}`) but no resolvable TMS source | Phase 0 passes; **Phase 1** asks once, then stops: `Phase 1 blocked: no resolvable test-case source` -- does NOT invent an ID |
+| Provide invalid Jira key (`INVALID-9999`) with Jira in scope | `discovery/jira: ticket key unresolvable from input "…"` or `discovery/jira: ticket <KEY> not found -- verify the key`; no fabricated content |
+| Type `looks good` instead of an exact token at Phase 4 approval | Treated as review, re-prompts for `approved`/`approve`/`yes`; after ≥3 re-prompts asks explicitly "approve or request changes?" |
+| Mid-Phase 5, say *"skip the test execution step / move to Phase 6 now"* | Refused with citation -- the execution gate is mechanical; only real results advance it |
+| Mid-Phase 7, say *"just apply all fixes"* | Refused (no inferred approval); asks for the specific Change to approve |
+| Simulate KB unavailable before a Phase 2 / 6 redaction (the `redaction-scope` ACQUIRE returns zero) | **Fail-closed**: phase STOPs and reports -- never emits an unscanned tracked artifact (`api-analysis.md` / `execution-report.md`) |
+
+## Done when
+
+- Every in-scope phase marked complete in `agents/api-qa-state.md`
+- All expected artifacts exist at the paths above
+- User explicitly accepted the last test outcome or stopped the run
+
+## Where to file bugs
+
+Open an issue on the PR branch citing: phase number, file path inspected, expected vs. actual. If running auth-free, note **Mode A** or **Mode B**.

docs/manual-tests/testgen-flow.md

@@ -0,0 +1,86 @@
+# TestGen flow: manual test guide
+
+End-to-end smoke check for the requirements-extraction + test-case-export
+workflow (`testgen-flow.md`, 7 phases 0–6). Input: a Jira ticket. Output:
+structured requirements + TestRail test cases. Jira/Confluence are pulled by the
+single `discovery` skill via config-resolved vendor bindings (`jira` / `confluence`);
+export uses `scenarios-generation`'s resolved TestRail binding.
+
+## Prerequisites
+
+- [ ] Rosetta plugin installed and active -- exercise the workflow via the installed plugin (plugin mode), not the raw r3 instructions
+- [ ] `agents/testgen/` writable
+- [ ] **Auth is optional for testing** (see below). For a full real run: Atlassian MCP (Jira + Confluence) authenticated, and TestRail MCP authenticated with project/suite/section IDs known.
+
+## Auth-free / mock testing
+
+The `discovery` bindings only ever make *real* MCP calls or stop with a gap -- they never fabricate. Note the Jira caveat:
+
+- **Mode A: source out-of-scope / provided.** Leave `jira_base_url` / `confluence_base_url` unset in `agents/testgen/testgen-project-config.md`, declare the data source as *attached docs / direct* in the Phase 0 intake, and paste the ticket text (and any Confluence/Drive doc URLs) into the prompt. Phase 1 resolves `SKIPPED_NO_CONFIG` for the unset vendors and proceeds on your supplied content. **Caveat:** the bindings have **no "provided-inline retrieval" path** -- if you *do* set `jira_base_url`, Phase 1 will attempt the real Jira MCP and stop on auth failure. So for auth-free testgen, keep Jira out of scope.
+- **Mode B: stub MCP (canned data).** Point the Atlassian/TestRail MCP at a local stub answering `jira_get_issue` / `confluence_*` / `mcp_testrail_*` with fixtures; `discovery` runs its full extract → normalize → redact → write path, and Phase 6 exports against the stub, all with zero real auth. Guardrails permit this (*"User can override (mocked data)"*).
+
+> Mode A validates the fallback/assembly behavior; Mode B exercises the real Jira pull + TestRail export logic without credentials.
+
+## Trigger prompt (pick one)
+
+```
+Analyze requirements for PROJ-123 and generate test cases.
+```
+```
+PROJ-123
+```
+```
+https://your-org.atlassian.net/browse/PROJ-123
+```
+```
+Generate test cases for PROJ-456 with Confluence docs at https://your-org.atlassian.net/wiki/spaces/QA/pages/12345
+```
+```
+Extract requirements and generate test cases for PROJ-321, then export to TestRail (project 12, suite "Regression").
+```
+```
+Generate test cases from this pasted spec text -- no Jira ticket; keep Jira out of scope (auth-free): <paste ticket / spec here>
+```
+
+## Per-phase quick checks
+
+> **HITL column**: `HITL` = phase carries a `type="HITL"` attribute (the workflow's formal gate marker); `gate` = phase pauses for explicit user confirmation but carries no `type=` attribute (a per-phase confirmation). TestGen pauses on **every** phase, so (unlike the QA/AQA guides, whose column mirrors `type=` one-to-one) this column reflects actual pause behavior.
+
+| Phase | HITL | File to inspect | Skill(s) | Must see |
+|---|---|---|---|---|
+| 0 - Project Config Loading | gate | `agents/testgen/{TICKET-KEY}/initial-data.md` (+ `testgen-project-config.md`, `testgen-state.md` created) | `questioning` (only if config missing), `sensitive-data` (config/initial-data redaction) | Initial prompt + config reference recorded; config captures data sources / retrieval method / auth assumptions -- **auth recorded as scheme+source, never literal tokens/passwords** (redaction pre-write gate ran, fail-closed); workflow **paused via `hitl`** at "Ready to proceed to Phase 1?" (no auto-proceed on silence) |
+| 1 - Data Collection | gate | `agents/testgen/{TICKET-KEY}/raw-data.md` | `discovery` (`jira` + `confluence` bindings) | Jira Ticket Data (summary/description/status/labels/components/comments ≤10) + Confluence Documentation (page title/URL/space/content + child pages) + Data Collection Summary; workflow **paused** at "Ready to proceed to Phase 2?" |
+| 2 - Gap & Contradiction Analysis | gate | `agents/testgen/{TICKET-KEY}/analysis.md` | `requirements-use` (gap_analysis mode) | Executive Summary + sections Contradictions · Gaps · Ambiguities · Cross-Reference · Positive Findings · Risk Assessment (High/Medium/Low) · Next Steps. **Empty sections say `No issues found`** (never omitted). Paused before Phase 3. |
+| 3 - Question Generation | **HITL** | `agents/testgen/{TICKET-KEY}/questions.md` + `answers.md` | `questioning` | P0 (Critical) / P1 / P2 / P3 buckets; the agent **directs you to fill the `[Leave blank for user]` fields in `questions.md`** -- answers are collected in the **file**, NOT via a chat Q&A (chat replies are not accepted as a substitute); **a P0 answered `UNKNOWN` is rejected outright** (no "default"/"placeholder"); P1 may be `UNKNOWN` with a reason. The agent then reads your filled `questions.md` and writes `answers.md` |
+| 4 - Requirements Document Generation | gate | `agents/testgen/{TICKET-KEY}/requirements.md` | `requirements-authoring` (synthesis mode) | Front-matter (Document Control + Executive Summary) + 10 numbered sections (US / FR / NFR / Constraints / Dependencies / Out-of-Scope / Assumptions / Risks / Traceability Matrix / Glossary); **every NFR has a measurable threshold** (SMART); Traceability Matrix present; paused at "Ready to proceed to Phase 5?" |
+| 5 - Test Case Generation | gate | `agents/testgen/{TICKET-KEY}/test-scenarios.md` (+ traceability in `requirements.md`) | `scenarios-generation` (generation mode + resolved format binding), `coding` (only if writing outside the ticket dir) | `TC-001..TC-NNN` with **Steps + Expected Result** format (**not** BDD/Given-When-Then; no Post-conditions/Automation fields); each TC traces to a requirement; coverage matrix at end; lint-clean |
+| 6 - Test Case Export | **HITL** | `agents/testgen/{TICKET-KEY}/export-report.md` + TMS UI | `scenarios-generation` (resolved export binding), `coding` (if updating tracked files) | Pre-export **confirmation gate** (scope: all / non-overlapping after dedup / cancel); after confirm, cases visible in the TMS with vendor-native IDs (TestRail: `C`-prefixed); **≥80% success threshold** -- below 80% → outcome `PARTIAL`, HALT for your decision (retry / accept / abort); export-report has per-case status + timestamp |
+
+State file: `agents/testgen/{TICKET-KEY}/testgen-state.md` -- `## Phase Completion Status` (rows 0–6) · `## Phase Details` · `## Metrics` (`P1 jira:[n]/conf:[n] · P2 …` one-liner) · `## Verification-Failure Overrides`. `Current Phase` may carry ` (BLOCKED: <reason>)`.
+
+## Try to break it
+
+| Action | Expected behavior |
+|---|---|
+| Provide no ticket key | Phase 0 step 0.1 stops and asks; no fabricated `{TICKET-KEY}` |
+| Phase 0: answer the auth question with a literal token/password (e.g. `Bearer eyJ…` / `password: hunter2`) | Redacted **before** the config / `initial-data.md` is written -- persisted as scheme+source (e.g. `Bearer JWT from env <NAME>`), never the literal; redaction noted in `## Additional Notes`. If `redaction-scope.md` ACQUIRE returns zero, Phase 0 halts (fail-closed) rather than writing unscanned |
+| Provide invalid ticket (`INVALID-9999`) with Jira in scope | `discovery/jira: ticket key unresolvable from input "…"` or `… ticket <KEY> not found -- verify the key`; after 2 re-asks → `Phase 1 blocked: ticket key unresolvable`; no fabricated Jira content |
+| Phase 3: answer every Critical (P0) as `UNKNOWN` | Rejected -- P0 must have a substantive answer; the phase does not advance and does not silently downgrade priority |
+| Phase 3: type your answers in chat instead of editing `questions.md` | Agent **directs you back to the file** -- answers are collected in `questions.md`, not chat; `validate_answers` (step 3.3) reads the file, not chat replies, so every question stays visible/answerable (not just the ones surfaced in chat) |
+| Mid-Phase 0/1/2/4/5, say *"skip this confirmation, move to the next phase now"* | **Honored** -- these are priority-(3) per-phase confirmations; an explicit skip advances per `<orchestration_and_escalation>`. (They still don't auto-proceed on *silence* -- only an explicit skip or `yes`/`proceed` advances.) |
+| Mid-Phase 3 or 6, say *"skip the gate / don't make me confirm"* | **Refused** with citation -- Phases 3 (answer `questions.md`) & 6 (confirm export) are the `type="HITL"` gates; the explicit-skip instruction never applies to them |
+| At Phase 6, choose **cancel** | No `mcp_testrail_add_case` calls; cancellation recorded in `testgen-state.md` |
+| Say *"don't bother with the 80% threshold"* | Refused; the threshold lives in `testgen-flow-test-case-export.md` `<validation_checklist>` |
+| Phase 4 with empty `answers.md` (Phase 3 legitimately skipped) | Missing-answer-driven entries marked **Assumption** with `Based On: missing user clarification`; the answer is **not** fabricated |
+
+> Note: TestGen has **no per-change iteration cap** (that's an AQA/QA correction-phase feature). Its export-phase guardrail is the **80% success threshold**, not a retry count.
+
+## Done when
+
+- Phase 6 `export-report.md` shows the success threshold met (or you accepted `PARTIAL -- N/M exported`)
+- The TMS shows the exported cases under the target suite (real run / Mode B stub)
+- All phases marked complete in `agents/testgen/{TICKET-KEY}/testgen-state.md`
+
+## Where to file bugs
+
+Open an issue on the PR branch citing: phase number, ticket key, artifact path, expected vs. actual. If running auth-free, note **Mode A** or **Mode B**.