fix: add --dangerously-skip-permissions for CI/CD environments#3
Merged
melanie531 merged 1 commit intoaws-samples:mainfrom Mar 19, 2026
Merged
Conversation
In CI/CD environments (CodeBuild, GitHub Actions, etc.), Claude Code has no user configuration (~/.claude/settings.json). Without permission bypass, Claude Code prompts for confirmation before executing Bash commands, which in -p (print) mode causes the tool call to be skipped. This results in: - Agent never executing skill scripts (e.g., python3 scripts/check.py) - Functional scores being artificially low (agent can't use skill tools) - Identical with-skill and without-skill results Fix: Add --dangerously-skip-permissions to both with-skill and without-skill Claude Code invocations. This is safe because: 1. Eval runs in isolated temp workspaces (no access to real data) 2. The agent is already sandboxed by --allowedTools 3. CI/CD environments are ephemeral containers This matches the Claude Code documentation recommendation for sandboxed environments without internet access.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In CI/CD environments (CodeBuild, GitHub Actions, etc.), Claude Code has no user configuration (
~/.claude/settings.json). Without permission bypass, Claude Code prompts for confirmation before executing Bash commands.In
-p(print) mode, this causes tool calls to be silently skipped — the agent never executes Bash commands likepython3 scripts/check.py, resulting in:Evidence
Same skill (
acme-compliance) tested in two environments:~/.claude/settings.json)skipDangerousModePermissionPrompt: trueThe 59-point gap is entirely explained by Claude Code's inability to execute Bash commands in the CI/CD environment.
Fix
Add
--dangerously-skip-permissionsto both_build_cmd_with_skill()and_build_cmd_without_skill()inClaudeRunner.This is safe because:
--allowedToolsTesting