Add a field-test harness for the AI-agent install prompt#47
Open
ejntaylor wants to merge 1 commit into
Open
Conversation
|
Introduces a comprehensive AI-agent field-test harness with clear docs and modular code. 🎯 Quality: 75% Good · 📦 Size: Large — consider splitting if possible 🛡️ Standards: no pre-flight fit check ran for this change — wire 📈 This month: Your 218th PR — above team average · Averaging Good |
Institutionalizes the ad-hoc setup used to find and fix the prompt's refusal modes and the guide's UX gaps: run a real agent through the full install in a throwaway vibe-platform fixture against a mocked Patchstack API, then score the outcome on 8 deterministic checks (install, single provisioning, hooks, widget + token match, claim URL surfaced, no production leak). - field-test/run.mjs — orchestrator: fixture + in-process mock + agent CLI (default headless Claude Code, pluggable via --agent-cmd) + scorecard; --rounds for stochastic agents, --prompt for variants, non-zero exit unless all rounds are fully green. Results (agent report, request log, scorecards) land in gitignored field-test/results/. - Mock endpoint is pinned via PATCHSTACK_ENDPOINT on the agent process, not a project file: every agent flagged a planted rc override as the top trust concern, and one deleted it and provisioned a real production site. Env pinning survives project edits and reads as platform plumbing. - personas/standard.md + personas/hostile.md — the second encodes the WebContainer/Bolt refusal pressure (supply-chain framing, no-authorization- URLs policy, sandboxed verification distrust) so that refusal mode stays a regression test. prompt.txt carries the harness-validated one-line prompt. - stub-compliant.mjs — scripted non-AI agent that self-tests the harness in ~1 min (verified 8/8 green; refusal path verified detected and non-green). - README points prompt editors at the harness before shipping changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ejntaylor
force-pushed
the
field-test-harness
branch
from
707160b to
8a00dd8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns the ad-hoc setup that found the prompt's four refusal modes into a repeatable dev harness under
field-test/(dev-only, not shipped in the npm package): it runs a real AI agent through the full install flow in a throwaway Lovable-style fixture against a mocked Patchstack API, and scores the outcome on 8 deterministic checks (install, provisioned exactly once, build hooks, widget present + userToken matches, claim URL surfaced to the user, no production leak).run.mjsorchestrates fixture + mock + agent CLI (default headless Claude Code, pluggable via--agent-cmd), supports--roundsfor stochastic agents and--promptfor testing variants, and exits non-zero unless every round is fully green; agent reports, request logs, and scorecards land in gitignoredfield-test/results/.PATCHSTACK_ENDPOINTon the agent's environment rather than a planted rc file — in earlier runs every agent flagged the rc override as the top trust concern, and one deleted it and provisioned a real production site.standard(Lovable-style platform agent) andhostile, which encodes the real Bolt/WebContainer refusal pressure so that refusal mode stays a regression test;prompt.txtcarries the harness-validated one-line prompt and the improve-and-retest loop is documented infield-test/README.md.stub-compliant.mjsis a scripted non-AI agent for ~1-minute harness self-tests.Test plan
npm install, mock provisioning, hooks, widget, claim URL).REFUSED, and the run exits non-zero.npm testuntouched (nosrc/changes in this PR).🤖 Generated with Claude Code