[Feature Request] Cross-artifact quality review between propose/ff and apply

[shohu]

Problem

When artifacts are generated in one pass via /opsx:propose or /opsx:ff (proposal → specs → design → tasks), they are built in forward dependency order. This means reverse feedback doesn't happen — issues that would surface when writing tasks don't propagate back to specs, and scope contradictions between artifacts go undetected.

This applies to both commands equally: /opsx:propose (core profile, generates all artifacts in one step) and /opsx:ff (expanded profile, fast-forwards through all planning artifacts). Both produce a complete artifact set in a single pass, so both share the same gap.

In practice, I consistently find these problems before /opsx:apply:

1. Cross-artifact contradictions — A feature included in proposal scope appears in design's Non-Goals
2. Specification gaps — Fallback behavior defined only in design but missing from spec (where it should be normative)
3. Duplication with existing specs/components — Tasks that reimplement functionality already available in existing specs or shared components
4. Missing design abstractions — Implementations that should use design patterns for maintainability (and AI comprehensibility) are specified as one-off implementations
5. File bloat in task design — Tasks that pack too many responsibilities into single files

Why single-pass generation can't solve this

• Human edits create drift — After exploring ideas and updating individual artifacts, cross-artifact consistency breaks
• Scope changes don't propagate — When a decision changes during discussion, not all artifacts get updated
• Semantic duplication is invisible to format validation — openspec validate catches structural issues but can't detect that a task duplicates an existing spec's capability

Concrete examples

After running /opsx:propose or /opsx:ff on real projects and reviewing artifacts before apply:

Priority	Pattern	Example
P0	Cross-artifact scope contradiction	Feature X listed in proposal scope but marked as Non-Goal in design
P0	Spec gap	Fallback behavior defined only in design, not in spec (where it should be normative with scenarios)
P1	Duplication with existing specs	Task reimplements capability already covered by an existing shared component
P1	Missing abstraction	Similar logic across multiple tasks should use a design pattern but is specified as separate implementations
P1	File bloat	Task design packs excessive responsibilities into single files
P2	Thin scenarios	Failure/edge-case THEN clauses lack specific assertions

None of these were caught during generation — they only surfaced through a separate review pass.

Relationship to existing features

openspec validate --strict: Catches structural/format issues (missing sections, malformed artifacts). Cannot detect semantic contradictions between artifacts or duplication with existing specs.

/opsx:clarify (PR #702): Resolves ambiguity within a single artifact through Q&A. The gap described here is across artifacts — checking consistency between proposal, specs, design, and tasks as a whole. These are complementary:

/opsx:propose or /opsx:ff
  → /opsx:clarify (resolve ambiguity within each artifact)
  → ★ gap (cross-artifact consistency + duplication analysis)
  → /opsx:apply

What I'm doing today

I've built a custom skill that runs as a review pass after /opsx:propose or /opsx:ff:

• Reads all artifacts (proposal/specs/design/tasks) + existing specs
• Checks cross-artifact scope consistency
• Analyzes duplication with existing specs and shared components (reuse / extend / new-shared / new-dedicated)
• Reports issues with P0/P1/P2 priority
• After user approval, updates artifacts and runs openspec validate --strict

This creates a natural review-then-approve flow before any modifications are made.

Discussion

I'd like to discuss whether this capability belongs in OpenSpec and, if so, what form fits best:

• A) New skill — e.g. /opsx:refine in the expanded profile
• B) Extend openspec validate — Add a semantic check layer alongside the existing structural checks (no new skill needed)
• C) Something else — Open to maintainer perspective on what fits the architecture

I'm aware of the concern about skill count growth. Happy to adapt the approach to whatever form works best for the project.

[demianmnave]

@shohu I'd be interested in testing out your artifact review skill if you have made it publicly available somewhere.

[shohu]

@demianmnave Thanks for your interest! Here's the skill I'm using:

openspec-refine-SKILL.md — Cross-artifact quality review skill for Claude Code

How to use it

1. Save the file as ~/.claude/skills/openspec-refine/SKILL.md (global) or .claude/skills/openspec-refine/SKILL.md (per-project)
2. After running /opsx:propose or /opsx:ff, invoke it with: "refine my change" or "review spec quality"

What it does

• Step 1: Collects all change artifacts + existing specs
• Step 2: Checks cross-artifact consistency (normative language, scenarios, acceptance criteria, hardcoded values, etc.)
• Step 2.3: Best practice validation — web-searches industry patterns and compares against each design Decision
• Step 2.5: Commonality analysis — checks for reuse opportunities with existing specs/components
• Step 3: Presents prioritized issues (P0/P1/P2) for user approval
• Step 4: Updates artifacts after approval
• Step 5: Runs openspec validate --strict

It's designed to catch the exact problems described in this issue: cross-artifact contradictions, spec gaps, duplication with existing components, missing design abstractions, and file bloat.

Feedback welcome — happy to iterate on this.

[demianmnave]

@shohu Awesome, thank you.

After a quick look, only a couple of things jump out at me. (Full disclosure, I am a bona fide noob at bending LLMs to my will, so please feel free to call me out on these suggestions.)

• CLAUDE.md and Claude skills are explicitly referenced, but it should be possible to use other models with this. For example, I use opencode pretty regularly and cycle through different models depending upon the needs of the change I am working on.
• I think 2.3 could result in heavy token usage because the instructions in 2.3.1 and the characteristics to compare the design decisions against in 2.3.3 seem too general. Maybe 2.3 could be (optionally) human guided to more quickly zoom in on relevant industry standards, design patterns, etc. For example, perhaps the model could be prompted to seek human intervention if the search terms that result from 2.3.1 aren't specific enough to narrow down the research inputs for 2.3.3.

I'll give this a try soon and let you know how it goes!

[shohu]

@demianmnave Great feedback — I've updated the Gist based on your points:

Changes made:

1. Model-agnostic usage section — Added a "Usage" section at the top explaining this works with any LLM tool (opencode, Cursor, Aider, etc.), not just Claude Code. Includes setup instructions for different tools.

2. Web search fallback (Step 2.3.2) — If web search isn't available in your environment, the skill now explicitly falls back to model knowledge and notes it for the user to verify independently.

3. Cross-artifact scope consistency check (Step 2) — Added a dedicated section that directly checks the P0 patterns from this issue: proposal scope vs design Non-Goals contradictions, normative behavior in design missing from spec, and scope boundary consistency across all artifacts. This was implicit before — now it's an explicit checklist item.

Updated Gist: https://gist.github.com/shohu/a219c8137f2967d720552a304badeeee

Looking forward to hearing how it goes when you try it!

[demianmnave]

@shohu For the change I'm working on now, Claude Sonnet 4.5 didn't propagate design.md updates to specs and tasks. I suspect that the requirement to propagate changes needs to be made more explicit in Step 4---reading this section myself, it's not immediately obvious.

The other instructions are generating interesting results, though. For example, the commonality analysis picked up on a function parameter pattern that I thought I'd made explicit, but didn't actually make it into the design. The current design comparison was also useful as it made explicit the alternatives I'd previously rejected---always useful for future me and future agents.

All in all, so far so good!

[shohu]

Thanks for the feedback @demianmnave! Great to hear the commonality analysis and design comparison are proving useful.

For the propagation issue — I've added an explicit Cross-artifact propagation (MUST) rule to Step 4 of the refine skill:

When updating design.md, check whether the changes affect spec.md or tasks.md — and propagate accordingly. After editing any artifact, explicitly verify that dependent artifacts are still consistent.

Updated in the gist. That said, the same gap likely exists in propose and apply-change steps too — I'll review those as well. Let me know if this helps with Sonnet 4.5 or if stronger guardrails are needed.

[demianmnave]

@shohu I can confirm your current version coerced Sonnet 4.5 to automatically update the artifacts for my current change. Thanks for the quick update!

I'll let you know how it goes as I apply it to more changes.

[shohu]

Glad to hear it worked! Looking forward to your feedback as you apply it to more changes.

[demianmnave]

@shohu I have been using openspec-refine with a greenfield project, and one thing that has come in handy is having the LLM record a markdown "scratchpad" as we work through discovered issues. It acts as a sort of agent memory, but is more targeted, and it is intended to be deleted after each change is completed.

I don't have enough agentic dev or tool experience to know if this is worth adding specifically to openspec-refine but thought I'd share anyway.

[shohu]

Thanks for sharing! The scratchpad pattern really resonates — it sits in a nice sweet spot between conversational memory and persistent state management.

Intentionally deleting it after each change is a smart move. It avoids the "stale context" problem where old notes end up misleading the agent in later sessions.

A couple of questions if you don't mind:

1. Does the LLM naturally write to the scratchpad, or do you explicitly prompt it?
2. Is it one scratchpad per refine session, or one per discovered issue?

This could work well as an optional mode in openspec-refine — something like a --scratchpad flag that auto-creates and cleans up. Would you be open to sharing a sample of what your scratchpad looks like?

[demianmnave]

@shohu I made an experimental opencode command (here) that wraps openspec-refine and encodes the format.

[shohu]

This is great, thanks for sharing the implementation! The structured format with priority levels (P0/P1/P2), status tracking (Open → Needs refinement → Consistent), and the constraint/decision log is really well thought out.

A few things I like:

• The explicit status legend tied to artifact state, not implementation state — that's a clean separation
• "Current Working Constraints / Decisions" as a running section — this captures context that usually gets lost between refine iterations
• The scratchpad being scoped per-change rather than per-session

I'd like to consider incorporating this into openspec-refine as a PR. Would you be okay with me referencing your command as prior art?

[demianmnave]

Absolutely.Could you post a link to your version so I can follow any improvements you make? Also, full disclosure, the scratchpad layout was arrived at while iterating with GPT 5.2 Codex and GPT 5.3 over a couple of changes.

I forgot to answer this:

Does the LLM naturally write to the scratchpad, or do you explicitly prompt it?

I initially prompted the LM through the scratchpad update process, then captured the core prompts and constraints that worked for me in that command I linked to. The command has so far lead the LLMs I use to automatically update the scratchpad while addressing refinement issues. (GPT 5.2 Codex and GPT 5.3)

I do prompt the LLM to address refinement issues one at a time so I can review the results. This iteration could help "remind" the LLM to perform the scratchpad update. The scratchpad is as much for me as for the LLM, though, so this is fine for my process. It would be interesting to see if an LLM maintains the scratchpad on its own if prompted to one-shot the refinement issues without human intervention.

One more detail: After each refinement issue is addressed to my liking, I commit the scratchpad and updated artifacts myself. This serves as a formal history of how we converged on the specs and tasks used to implement the change.

[shohu]

Thanks! I'll share the link once the PR is up.

Interesting that the layout was arrived at through iteration with GPT 5.2/5.3 — if the pattern works across models, that's a good signal for generalizability.

The commit-per-issue approach is great. Having the scratchpad in git history gives you traceability for "why did this spec end up this way." I'd like to include that pattern when incorporating this into openspec-refine.

I'll also experiment with whether the LLM maintains the scratchpad on its own in a one-shot refinement flow.