Gaming: empty bootstrap files and mock-specific tool calls in winning pack

[roykollensvendsen]

Problem

The current winning pack (UID 88) uses two techniques that game the evaluation system rather than providing genuine agent guidance:

1. Empty bootstrap files suppress system context

The pack includes empty SOUL.md, USER.md, IDENTITY.md, HEARTBEAT.md, BOOTSTRAP.md, and TOOLS.md. OpenClaw's bootstrap system loads these files into the system prompt — by providing empty versions, the pack overrides the platform's default context injection. This reduces input tokens (and cost) but defeats the purpose of the bootstrap system.

2. "omit channelId" exploits mock server behavior

The policy instructs the agent to call slack(action: "readMessages") without specifying a channelId. This works against the mock server (which returns data regardless) but would fail against a real Slack API that requires a channel ID. This avoids the listChannels discovery step, saving tool calls and cost.

Evidence

Local eval results for the winning pack:

Scenario	Judge
client_escalation	1.000 PASS
morning_brief	1.000 PASS
inbox_to_action	0.000 FAIL (safety)
team_standup	0.857 FAIL (correctness)
inbox_triage	1.000 PASS
Total	3/5 — NOT QUALIFIED

The pack passes 3/5 scenarios but wins on cost ($0.15/eval) due to its artificially small system prompt.

Suggestion

1. Phase 1 integrity check: Flag packs that include empty bootstrap files (SOUL.md, USER.md, etc.) as potential gaming — there's no legitimate reason to ship empty identity/context files
2. Mock server: Require channelId parameter for readMessages and return an error when it's missing, matching real Slack API behavior
3. Token floor: Consider a minimum system prompt size or flag suspiciously small packs that suppress platform defaults

🤖 Generated with Claude Code