Skip to content

penacristian/coding-agent-reliability

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

coding-agent-reliability

Two small artifacts for making LLM coding agents trustworthy: a guardrail protocol that stops the agent from fabricating on the way in, and an adversarial reviewer that catches the confident-but-wrong claim on the way out.

Both are plain markdown. Drop them into any agent harness (Claude Code, or anything that reads agent instructions). No dependencies.

The idea

LLM coding agents fail in two recurring ways, and almost everyone tries to fix both by asking the model to "be more careful." That doesn't work, because the model doesn't know it's being careless.

So instead of asking for care, encode it:

  1. On the way in — agents hallucinate functions, import symbols they never verified, and declare code "unused" without checking. Each of these is a named, enumerable failure mode. The Anti-Hallucination Protocol turns each one into a mechanical rule that ends in a replayable verification step. Cite with file:line and a verbatim quote or say you didn't verify. Grep for a symbol before importing it. Include the grep command when you claim something is dead code.

  2. On the way out — a reviewer (agent or human) declares a change correct, and the confidence is wrong. The adversarial-reviewer does not re-review the diff. It takes the reviewer's claims ("this can only be redeemed once", "this migration is reversible") and tries to falsify each one, with a sub-2-minute verification step per challenge. It's allowed to return CLEARED, so it doesn't manufacture concerns to look useful.

The non-obvious move is the second one: most setups add a second reviewer that re-reads the same code in parallel. Attacking the first reviewer's claim instead catches a different class of bug — the race condition or missing authorization check that's invisible when you read the code top-to-bottom but obvious the moment you attack the sentence "this is correct." See the worked example.

What's here

File What it is
guardrails/anti-hallucination-protocol.md 7 rules, each a forcing function for one LLM failure mode. Reference it from your agent instructions.
agents/adversarial-reviewer.md A subagent that red-teams a reviewer's claim of correctness. Claude Code agent format; the prompt body adapts to any harness.
examples/worked-example.md An adversarial pass catching a concurrency race the original review missed.

How to use it

Claude Code: copy agents/adversarial-reviewer.md into .claude/agents/, and reference guardrails/anti-hallucination-protocol.md from your CLAUDE.md. Your code-review agent calls adversarial-reviewer with its claims as a final pass before commit.

Any other harness: the files are just prompts. Paste the protocol into your system instructions; run the adversarial reviewer as a second pass that receives {change summary, reviewer's claims, files touched} and returns challenges.

Optional: enforce the adversarial pass as a pre-commit step for changes touching auth, migrations, concurrency, money, or public APIs.

Adapting it

The only stack-specific rule is "Build-or-flag" — swap in your build / type-check / test command. Everything else is language-agnostic. The attack playbooks in the adversarial reviewer are a starting set; add the failure modes specific to your domain.

License

MIT. Use it, fork it, sell what you build with it.

About

Anti-hallucination protocol + an adversarial reviewer that red-teams a reviewer's claim of correctness. Markdown, drop-in for any LLM coding-agent harness.

Resources

License

Stars

12 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors