patch: Add evals category for agent / workflow projects (closes #44)#46
Merged
Conversation
This was referenced May 6, 2026
Closes #44. The agent and workflow templates ship a full eval harness under `evals/` (assertions, discovery, mock_factory, runner, package init, README). None of those files were covered by any patch category, so updates were invisible to `fips-agents patch check`. This adds an `evals` category to AGENT_FILE_CATEGORIES covering just the harness machinery and registers a `patch evals` subcommand. Set ask_before_patch=True since users may have customized the harness. User-authored eval inputs (`evals/evals.yaml` and `evals/fixtures/`) go to AGENT_NEVER_PATCH so the test plan and data fixtures stay under the user's control. Stacks on top of #43. Assisted-by: Claude Code (Opus 4.7)
rdwj
force-pushed
the
feat/44-evals-patch-category
branch
from
7baed48 to
13fc468
Compare
rdwj
added a commit
that referenced
this pull request
May 7, 2026
- Add v0.12.0 changelog entry (manifest loader, evals category, MCP claude category, never-patch matcher fix, pattern gap fills). - Update Patch Commands section: list .fips-template.yaml manifest support, add Gateway/UI category table, refresh per-type tables to match the actual category surface after #43, #46, #48, #49. - Expand the user-customized-files paragraph to cover the new AGENT_NEVER_PATCH entries and the gateway/UI never-patch list. Assisted-by: Claude Code (Opus 4.7)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 2 of #42 — adds an
evalscategory toAGENT_FILE_CATEGORIES.Stacked on top of #43; please merge that first or the diff in this PR will be noisy.
What
evalscategory coversevals/__init__.py,evals/assertions.py,evals/discovery.py,evals/mock_factory.py,evals/run_evals.py,evals/README.md.ask_before_patch=Truesince users may have customized the harness.evals/evals.yaml(the test plan) andevals/fixtures/**(test data) go toAGENT_NEVER_PATCH— those are user-authored.patch evalssubcommand wires through the existing_patch_categoryhelper.CLAUDE.mdwith the new agent/workflow category list.Out of scope
Per-template
.fips-template.yamlmanifests — tracked in #45.Test plan
TestEvalsCategorycovering category placement, pattern coverage, ask_before_patch behavior, and never-patch entries for user inputs.black src testsclean.ruff check src testsclean.