[prompt-analysis] Copilot PR Prompt Analysis - 2026-06-08

[github-actions[bot]]

Summary

Analysis Period: Last 30 days | Run: §27167547670

• Total PRs: 1,000 | Merged: 794 (79.9%) | Closed: 199 (19.9%) | Open: 7 (0.7%)
• Overall success rate is 79.9%, recovering from the 79.1% low on June 7
• Key finding: WIP-labeled PRs have a 27% success rate — the strongest negative signal in the dataset

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
📝 Documentation	35	30	5	86%
🧪 Testing	170	144	24	85%
✨ Feature	343	288	53	84%
🔀 Other	179	145	33	81%
🐛 Bug Fix	247	172	73	70%
🔧 Refactor	23	14	9	61%
❓ Empty Body	3	1	2	33%

Prompt Characteristics Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 187 words (shorter than closed PRs)
• Most common action verbs: add, update, remove, support, create, ensure, parse
• Conventional commit prefixes: fix: → 88% merged, feat: → 84% merged
• File/path references: 79% of merged PRs reference specific files

Example successful prompts:

1. PR Fix "Organizations" typo in docs sidebar (2 locations) #37934 (Merged): "Two adjacent sidebar labels in docs/astro.config.mjs misspelled 'Organizations'... One entry also had a double space." — specific file, clear problem, tight scope.

2. PR Replace codex-incompatible gpt-5-mini variant in Daily Cache Strategy Analyzer #37930 (Merged): "Daily Cache Strategy Analyzer was failing whenever experiment activation selected gpt-5-mini, because codex cannot run that model and exits with unknown-model/AI-credit errors." — exact error, named component, clear root cause.

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 244 words (31% longer than merged)
• WIP label: 55 PRs marked [WIP] → only 27% success rate (40/55 closed)
• Refactor-heavy PRs close at higher rates (61% success vs 84%+ for feature/testing)
• Vague problem statements without specific error messages or file references

Example unsuccessful prompts:

1. PR [WIP] Fix failing GitHub Actions job 'agent' #37890 (Closed): "Thanks for asking me to work on this... Fix the failing GitHub Actions job 'agent'. Analyze the Actions logs, identify the root cause of the failure..." — generic instruction with no specific failure detail.

2. PR Detect MaxAI credit-limit failures from firewall API-proxy event logs in conclusion handling #37927 (Closed): Prompt describes the correct problem in detail but the implementation path duplicated existing logic, leading to a superseding PR. Completeness ≠ correctness.

View Length Distribution Detail

Word Count Range	Merged Count	Closed Count
0–50 words	10	5
50–100 words	71	26
100–200 words	426	86
200–400 words	265	63
400+ words	22	19

Note: The 400+ word bucket has the worst merged-to-closed ratio — extremely long prompts correlate with more complex/risky changes.

Key Insights

• WIP is a red flag: PRs with [WIP] in the title have a 27% success rate vs the overall 80%. Copilot should complete work before opening a PR, not use it as a draft placeholder.
• Shorter prompts outperform longer ones: Merged PR prompts average 187 words vs 244 for closed — a 31% difference. Focused, scoped prompts succeed more reliably.
• Testing and documentation tasks are the most reliable: At 85–86% success rates, these categories outperform even feature work. Bug fix PRs (70%) and refactoring PRs (61%) see more churn, likely due to implementation disagreements or superseding fixes.

Recommendations

1. DO: Reference specific file paths, function names, or error messages in the prompt. Merged PRs with file references succeed at 79% vs similar specificity for closed at 76%.
2. DO: Use conventional commit prefixes (fix:, feat:). fix: prompts succeed at 88%, the highest of any naming convention observed.
3. DO: Keep prompts under 200 words for best results — 426/794 merged PRs (54%) fall in the 100–200 word range.
4. AVOID: Marking PRs as [WIP] — this signals incomplete work and predicts closure at 73% rate.
5. AVOID: Pure refactoring without a concrete outcome; refactor PRs close at 39%, vs 16% for feature PRs.

Historical Trends (Last 7 Days)

Date	PRs	Merged	Success Rate	Notable
2026-05-28	1,000	799	80.5%	—
2026-05-30	1,000	802	80.5%	—
2026-06-01	1,000	802	80.4%	—
2026-06-02	1,000	809	81.0%	Peak
2026-06-06	1,000	788	~79.1%	Dip
2026-06-07	1,000	791	79.1%	Low
2026-06-08	1,000	794	79.9%	↑ Recovery

Trend: Success rate dipped from the 81% peak on June 2 to 79.1% on June 6–7, with today showing a recovery to 79.9%. The long-run mean appears to be ~80%.

References: §27167547670

Generated by 🔍 Copilot PR Prompt Pattern Analysis · ⌖ 68 AIC · ⊞ 19.9K · ◷

• expires on Jun 9, 2026, 1:28 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-06-09T21:28:56.527Z.

Closed by Workflow