pdd bug agents still closes duplicated issues that are not resolved

[jiaminc-cmu]

For example, this one:#530;
I had to manually reopen it

[prompt-driven-github]

🚀 Job Queued!

Job ID: woFHoQx6PP6kc5lG8tKd
Triggered by: @jiaminc-cmu
Label: bug

View Live Progress
Model: pdd-sonnet (claude-sonnet-4-5-20250929)

[prompt-driven-github]

Step 1: Duplicate Check

Status: No duplicates found

Search Performed

• Searched for: "duplicate close", "bug agent close", "agent close", "incorrectly close", "wrongly close", "workflow close", "reopened"
• Issues reviewed: 50+
• Examined related issue: PDD bug hangs in after step 5 on duplicate bug #303 ("PDD bug hangs in after step 5 on duplicate bug")

Findings

This appears to be a new issue. While I found issue #303 which discussed the pdd bug workflow behavior on duplicates, it described a different problem (workflow hanging after detecting duplicates, which was later determined to be a network issue).

Issue #533 reports a distinct problem: the pdd bug agent is closing duplicate issues even when the original/canonical issue is not yet resolved. The user references issue #530 as an example that was incorrectly closed and had to be manually reopened.

This is a legitimate bug in the duplicate detection/handling logic. The agent should:

1. Not close a duplicate issue if the canonical issue is still open/unresolved
2. Only close duplicates when the original issue has been resolved

No other issues in the repository describe this specific behavior, so this is confirmed as a new issue requiring investigation.

---

Proceeding to Step 2: Documentation Check

[prompt-driven-github]

Step 2: Documentation Check

Status: Confirmed Bug

Documentation Reviewed

• README.md (lines 2186-2199): Bug workflow documentation
• prompts/agentic_bug_step1_duplicate_LLM.prompt (lines 31-48): Duplicate handling instructions
• tests/test_issue_469_duplicate_unresolved.py: Comprehensive test suite for this exact scenario
• Issue pdd-change should support documentation-only changes without requiring dev units #520 state: Verified as OPEN (unresolved)
• Issue pdd-change finished step 1-5 but failed at step 6 #530 state: Verified as CLOSED (incorrectly closed as duplicate)

Findings

This is a confirmed bug - a regression of issue #469.

What the documentation says:
The prompt file agentic_bug_step1_duplicate_LLM.prompt explicitly instructs the agent:

"If duplicates are found BUT the original issue is unresolved:

• Note the relationship to the original issue but DO NOT close this issue
• Report that the original was not resolved and this issue should proceed"

What actually happened:

1. Issue pdd-change finished step 1-5 but failed at step 6 #530 was correctly identified as a duplicate of issue pdd-change should support documentation-only changes without requiring dev units #520
2. Issue pdd-change should support documentation-only changes without requiring dev units #520 is still OPEN (I verified with gh issue view 520 - state: OPEN, closedAt: null)
3. Despite pdd-change should support documentation-only changes without requiring dev units #520 being unresolved, the agent closed pdd-change finished step 1-5 but failed at step 6 #530 with the comment "Closing this issue as a duplicate"
4. This violated the documented behavior in the prompt

Evidence:

• Issue pdd-change should support documentation-only changes without requiring dev units #520 title: "pdd-change should support documentation-only changes without requiring dev units"
• Issue pdd-change should support documentation-only changes without requiring dev units #520 state: OPEN (unresolved)
• Issue pdd-change finished step 1-5 but failed at step 6 #530 was closed by the bot as a duplicate even though pdd-change should support documentation-only changes without requiring dev units #520 remains unresolved
• This is exactly the bug that issue pdd sessions cleanup shows misleading success message when all cleanups fail #469 was supposed to fix, with comprehensive tests in test_issue_469_duplicate_unresolved.py

Root Cause Analysis

This appears to be a regression of the fix for issue #469. The prompt contains the correct instructions to check resolution status, but the agent is not following them correctly. The agent is still closing duplicate issues when the canonical issue remains open/unresolved.

---

Proceeding to Step 3: Triage

[prompt-driven-github]

Step 3: Triage

Status: Proceed

Summary

We have sufficient information to attempt reproduction of this bug. Here's what we understand:

Expected Behavior:

• When issue pdd-change finished step 1-5 but failed at step 6 #530 is marked as a duplicate of issue pdd-change should support documentation-only changes without requiring dev units #520, and issue pdd-change should support documentation-only changes without requiring dev units #520 is still OPEN (unresolved), issue pdd-change finished step 1-5 but failed at step 6 #530 should NOT be automatically closed
• This behavior is explicitly documented in the prompt files

Actual Behavior:

• Issue pdd-change finished step 1-5 but failed at step 6 #530 was closed as a duplicate of pdd-change should support documentation-only changes without requiring dev units #520, even though pdd-change should support documentation-only changes without requiring dev units #520 remains OPEN
• The user had to manually reopen issue pdd-change finished step 1-5 but failed at step 6 #530

Context:

• This is a regression of issue pdd sessions cleanup shows misleading success message when all cleanups fail #469, which was previously fixed
• The fix included updated prompts, logic to check resolution status, and a comprehensive test suite (test_issue_469_duplicate_unresolved.py)
• We have a concrete example to examine (issues pdd-change finished step 1-5 but failed at step 6 #530 and pdd-change should support documentation-only changes without requiring dev units #520)

Investigation Plan:

1. Examine the actual behavior that occurred with issues pdd-change finished step 1-5 but failed at step 6 #530 and pdd-change should support documentation-only changes without requiring dev units #520
2. Review the test suite to understand why it didn't catch this regression
3. Trace through the duplicate detection and closing logic
4. Identify why the fix for pdd sessions cleanup shows misleading success message when all cleanups fail #469 is not working as intended

---

Proceeding to Step 4: Reproduction

[prompt-driven-github]

Step 4: Reproduction

Status: Reproduced

Environment

• OS: Linux 6.9.12
• Repository: promptdriven/pdd (main branch, commit 043b1dc)
• Affected workflow: Bug investigation (agentic_bug_orchestrator.py)
• Test infrastructure: Tests exist for issue pdd sessions cleanup shows misleading success message when all cleanups fail #469 which was a previous fix for this same problem

Steps Taken

1. Examined issue pdd-change finished step 1-5 but failed at step 6 #530 which was incorrectly closed as a duplicate of pdd-change should support documentation-only changes without requiring dev units #520
2. Verified issue pdd-change should support documentation-only changes without requiring dev units #520 is still OPEN (unresolved) - created 2026-02-14, still open
3. Reviewed timeline events showing the bot closed pdd-change finished step 1-5 but failed at step 6 #530 on 2026-02-15T07:17:51Z
4. User had to manually reopen pdd-change finished step 1-5 but failed at step 6 #530 on 2026-02-15T07:24:57Z
5. Examined the Step 1 duplicate check prompt (pdd/prompts/agentic_bug_step1_duplicate_LLM.prompt)
6. Found the hard-stop logic in pdd/agentic_bug_orchestrator.py:441
7. Reviewed existing test suite for issue pdd sessions cleanup shows misleading success message when all cleanups fail #469 which was a previous fix for this same bug
8. Analyzed the actual Step 1 output from issue pdd-change finished step 1-5 but failed at step 6 #530's investigation

Observed Behavior

Issue #530 was incorrectly closed as a duplicate even though the canonical issue (#520) remains OPEN.

Step 1 output from issue #530:

## Step 1: Duplicate Check

**Status:** Duplicate of #520

### Search Performed
- Searched for: documentation changes, "no dev unit", "step 6", early exit issues
- Issues reviewed: 100+ bug-labeled issues and 120+ issues matching search terms

### Findings
This issue is a **duplicate of #520** (...)

### Action Taken
Closing this issue as a duplicate.

What should have happened:
According to the prompt at lines 45-48, when a duplicate is found BUT the original is unresolved, the output should be:

• Status: No duplicates found
• Note the relationship but DO NOT close the issue
• Report that the original was not resolved and this issue should proceed

What actually happened:
The LLM output "Status: Duplicate of #520" which triggered the hard-stop at line 441 of the orchestrator, causing the workflow to stop and the issue to be closed.

Key observations:

1. The Step 1 output did NOT mention checking the resolution status of pdd-change should support documentation-only changes without requiring dev units #520
2. No gh issue view 520 command appears to have been executed
3. The LLM violated the prompt instructions at lines 31-36 which require verifying resolution status
4. The hard-stop logic at line 441 correctly detected "Duplicate of #" in the output and stopped the workflow

Affected Files

• pdd/agentic_bug_orchestrator.py:441 - Hard-stop logic that triggers when "Duplicate of #" appears in Step 1 output:

  if step_num == 1 and "Duplicate of #" in output:
      msg = f"Stopped at Step 1: Issue is a duplicate. {output.strip()}"
      if not quiet:
          console.print(f"⏹️  {msg}")
      return False, msg, total_cost, last_model_used, changed_files

• pdd/prompts/agentic_bug_step1_duplicate_LLM.prompt:31-48 - Prompt instructions that require checking resolution status but were not followed by the LLM:

  3. **Verify original issue resolution status**
     For each potential duplicate, check if the original issue was actually resolved:
     - Use `gh issue view <number>` to check the original issue's state
     - Check the workflow state comment for step outcomes
     - An issue is considered **unresolved** if: it is still open, all workflow steps failed, or no fix was merged
     - An issue is considered **resolved** if: it is closed with a merged fix, or workflow steps completed successfully
  
  4. **Handle duplicates**
     If duplicates are found AND the original issue was **resolved**:
     - Close this issue with a comment linking to the original
     
     If duplicates are found BUT the original issue is **unresolved**:
     - Note the relationship to the original issue but DO NOT close this issue
     - Report that the original was not resolved and this issue should proceed
  

Notes

1. This is a regression of issue pdd sessions cleanup shows misleading success message when all cleanups fail #469: Comprehensive tests were written in tests/test_issue_469_duplicate_unresolved.py to prevent this exact bug, but the LLM is still not following the prompt instructions correctly.

2. The prompt fix from pdd sessions cleanup shows misleading success message when all cleanups fail #469 is in place: The prompt correctly instructs the LLM to check resolution status (lines 31-36) and handle unresolved duplicates appropriately (lines 45-48).

3. The orchestrator code is correct: The hard-stop logic at line 441 correctly detects when the LLM outputs "Duplicate of #" and stops the workflow.

4. Root cause: The LLM agent is not following the prompt instructions to:

   • Use gh issue view to check the resolution status of the original issue
   • Only output "Duplicate of #XXX" when the original is resolved
   • Output "No duplicates found" (with explanatory text) when the original is unresolved

5. The LLM continued past Step 1: Interestingly, looking at the full comment thread, the bot posted Step 2, Step 3, etc. comments on issue pdd-change finished step 1-5 but failed at step 6 #530, which suggests the workflow may have continued even after declaring it a duplicate. This is inconsistent with the hard-stop logic.

6. Possible explanation: The Step 1 comment was posted by the LLM agent using gh issue comment, and then the agent may have also executed gh issue close separately. The orchestrator code shows the workflow should stop at line 445 when "Duplicate of #" is detected, but the presence of Step 2+ comments suggests either:

   • The duplicate check happened in a separate job run
   • The hard-stop didn't work as expected
   • The issue was closed by a separate mechanism

---

Proceeding to Step 5: Root Cause Analysis

[prompt-driven-github]

Step 5: Root Cause Analysis

Root Cause Identified: Yes

Summary

The bug occurs because the orchestrator at pdd/agentic_bug_orchestrator.py:441 blindly trusts the LLM's duplicate detection without validating that the original issue is actually resolved. When the LLM outputs "Duplicate of #XXX" in Step 1, the orchestrator immediately triggers a hard-stop and exits the workflow, even if issue #XXX is still open/unresolved. This is a trust-but-don't-verify problem - the prompt correctly instructs the LLM to check resolution status before declaring duplicates, but the orchestrator provides no safety net when the LLM fails to follow these instructions.

Technical Details

• Location: pdd/agentic_bug_orchestrator.py:441
• Problematic Code:

# Step 1: Duplicate Check
if step_num == 1 and "Duplicate of #" in output:
    msg = f"Stopped at Step 1: Issue is a duplicate. {output.strip()}"
    if not quiet:
        console.print(f"⏹️  {msg}")
    return False, msg, total_cost, last_model_used, changed_files

• Why it fails: The hard-stop logic uses a simple string match ("Duplicate of #") without any validation. It doesn't:

  1. Extract the duplicate issue number
  2. Check if that issue is actually closed
  3. Verify that the issue was resolved (not just closed without a fix)

  This means when the LLM makes a mistake (outputting "Duplicate of pdd-change should support documentation-only changes without requiring dev units #520" even though pdd-change should support documentation-only changes without requiring dev units #520 is still OPEN), the orchestrator has no mechanism to catch the error.

External Research

• Search terms: "LLM prompt adherence", "duplicate detection", "issue resolution status check"
• Sources checked: Research on LLM-as-judge evaluation, prompt testing tools, duplicate detection systems
• Relevant findings:
  • LLM-as-judge evaluations for prompt adherence show that monitoring completion scores and setting thresholds can alert when outputs don't follow instructions
  • Research on LLM interactions found that ineffective conversations contain knowledge gaps in prompts 54.7% of the time
  • Multiple systems (Linear, Elasticsearch, Dolomite AI) implement duplicate detection with LLMs but don't publicly document how they handle resolution status validation

Repository History

• Relevant commits:
  • 043b1dc (2026-02-14) - Initial codebase commit that includes the current prompt with resolution status checking instructions
  • The prompt at lines 31-48 has ALWAYS included instructions to check resolution status
• Related PRs:
  • No PRs found that modified the hard-stop logic at line 441
• Similar past issues:
  • Issue pdd sessions cleanup shows misleading success message when all cleanups fail #469 test file (tests/test_issue_469_duplicate_unresolved.py) documents a previous fix for duplicate detection
  • However, the test only validates that the PROMPT contains resolution-checking instructions (lines 331-405)
  • The test does NOT validate that the ORCHESTRATOR enforces these instructions

Experiments Performed

1. Examined actual failure case (issue pdd-change finished step 1-5 but failed at step 6 #530):

   • Step 1 comment posted at 07:17:35Z said "Status: Duplicate of pdd-change should support documentation-only changes without requiring dev units #520"
   • Issue pdd-change finished step 1-5 but failed at step 6 #530 was closed by bot at 07:17:51Z (16 seconds later)
   • Issue pdd-change should support documentation-only changes without requiring dev units #520 is STILL OPEN today (created 2026-02-14, state: "OPEN")
   • The LLM output "Duplicate of pdd-change should support documentation-only changes without requiring dev units #520" WITHOUT checking that pdd-change should support documentation-only changes without requiring dev units #520 was resolved
   • The orchestrator saw "Duplicate of #" and immediately stopped the workflow

2. Verified prompt instructions are correct:

   • Lines 31-36: Explicitly instructs to check resolution status using gh issue view
   • Lines 39-43: Only close if original is resolved
   • Lines 45-48: DO NOT close if original is unresolved
   • The prompt is comprehensive and clear

3. Confirmed no validation in orchestrator:

   • Line 441 only checks for string "Duplicate of #"
   • No code extracts the issue number
   • No code calls gh issue view to verify status
   • The orchestrator completely trusts the LLM output

Fix Location

• File(s) to modify: pdd/agentic_bug_orchestrator.py (and potentially the same logic in agentic_change_orchestrator.py and agentic_test_orchestrator.py)

• Why this location: The bug is NOT in the prompt (which is correct) - it's in the orchestrator's lack of validation. The orchestrator currently implements a "trust the LLM blindly" approach, but LLMs are not 100% reliable at following complex multi-step instructions. The fix must add a validation layer AFTER the LLM outputs "Duplicate of #" but BEFORE triggering the hard-stop.

• Existing pattern: The codebase already has patterns for validation:

  • File existence checks before operations
  • Git status verification before commits
  • Test result parsing and validation
  • However, NO validation exists for LLM outputs that trigger hard-stops

• Fix approach: Add validation logic at line 441:

  1. Extract duplicate issue number from the LLM output using regex: Duplicate of #(\d+)
  2. Verify the original issue is resolved by calling gh issue view <number> --repo {repo_owner}/{repo_name} --json state,closedAt
  3. Only trigger hard-stop if resolved: If the original issue is OPEN, log a warning that the LLM made an error, continue the workflow, and optionally post a clarification comment
  4. If resolved, proceed with hard-stop: Keep the current behavior for true duplicates

  This extends the existing hard-stop pattern by adding a guard condition that validates the LLM's judgment before acting on it.

  Example implementation:

  # Step 1: Duplicate Check
  if step_num == 1 and "Duplicate of #" in output:
      # Extract duplicate issue number
      import re
      match = re.search(r"Duplicate of #(\d+)", output)
      if match:
          duplicate_num = match.group(1)
          
          # Verify the original issue is actually resolved
          try:
              result = subprocess.run(
                  ["gh", "issue", "view", duplicate_num, "--repo", f"{repo_owner}/{repo_name}", 
                   "--json", "state,closedAt"],
                  capture_output=True, text=True, check=True, cwd=cwd
              )
              issue_data = json.loads(result.stdout)
              
              # Only stop if the original issue is closed (resolved)
              if issue_data["state"] == "CLOSED":
                  msg = f"Stopped at Step 1: Issue is a duplicate of #{duplicate_num} (resolved). {output.strip()}"
                  if not quiet:
                      console.print(f"⏹️  {msg}")
                  return False, msg, total_cost, last_model_used, changed_files
              else:
                  # Original issue is still open - LLM made an error
                  if not quiet:
                      console.print(f"[yellow]Warning: LLM claimed duplicate of #{duplicate_num} but that issue is still OPEN. Continuing workflow.[/yellow]")
                  # Continue to next step instead of stopping
          except (subprocess.CalledProcessError, json.JSONDecodeError, KeyError) as e:
              # If we can't verify, err on the side of caution and continue
              if not quiet:
                  console.print(f"[yellow]Warning: Could not verify duplicate status for #{duplicate_num}: {e}. Continuing workflow.[/yellow]")

---

Proceeding to Step 5.5: Prompt Classification

[prompt-driven-github]

Step 5.5: Prompt Classification

Classification: Code Bug

DEFECT_TYPE: code

Analysis

The root cause identified in Step 5 shows that the orchestrator code does not correctly implement the prompt specification. The prompt at pdd/prompts/agentic_bug_step1_duplicate_LLM.prompt (lines 31-48) explicitly instructs the LLM to verify resolution status and only output "Duplicate of #XXX" when the original issue is resolved. However, the orchestrator code at pdd/agentic_bug_orchestrator.py:441 blindly trusts the LLM's output without performing any validation.

Evidence

• Prompt specifies: Lines 31-48 of the duplicate detection prompt clearly state:

  • "Verify original issue resolution status" (line 31)
  • "Use gh issue view to check the original issue's state" (line 33)
  • "An issue is considered unresolved if: it is still open..." (line 35)
  • "If duplicates are found BUT the original issue is unresolved: DO NOT close this issue" (lines 45-46)

• Code implements: Line 441 simply checks for the string "Duplicate of #" and immediately stops the workflow:

  if step_num == 1 and "Duplicate of #" in output:
      msg = f"Stopped at Step 1: Issue is a duplicate. {output.strip()}"
      return False, msg, total_cost, last_model_used, changed_files

  This code performs no validation that the original issue is actually resolved.

• User expects: Duplicate issues should NOT be closed when the canonical issue is still unresolved (which matches what the prompt specifies).

Conclusion

The code does not correctly implement the prompt specification. The prompt requires resolution status validation, but the orchestrator code skips this validation step and blindly trusts the LLM output. This is a code implementation bug, not a prompt defect.

The fix should add validation logic at line 441 to:

1. Extract the duplicate issue number from the LLM output
2. Call gh issue view to verify the original issue's state
3. Only trigger the hard-stop if the original is actually closed/resolved
4. Log a warning and continue the workflow if the original is still open

---

Proceeding to Step 6: Test Plan

[prompt-driven-github]

Step 6: Test Plan

Existing Test Coverage

• Test file: tests/test_issue_469_duplicate_unresolved.py

• Current coverage: This test suite was created for issue pdd sessions cleanup shows misleading success message when all cleanups fail #469 (the original fix for this problem). It includes:

  • Unit tests for the _check_hard_stop functions in change and test orchestrators
  • Prompt regression tests to ensure prompts contain resolution status checking instructions
  • Integration tests verifying the bug orchestrator continues when duplicates are unresolved
  • Tests for conversational mentions of related issues without triggering hard stops

• Gap: The bug orchestrator itself lacks validation at line 441. The existing tests verify that:

  1. The prompt instructs the LLM to check resolution status ✅
  2. The change/test orchestrators have _check_hard_stop validation ✅
  3. End-to-end behavior works when LLM follows instructions ✅

  However, no test validates what happens when the LLM fails to follow the prompt and outputs "Duplicate of pdd-change should support documentation-only changes without requiring dev units #520" even though pdd-change should support documentation-only changes without requiring dev units #520 is still OPEN. The hard stop at line 441 of pdd/agentic_bug_orchestrator.py blindly trusts the LLM output without verification.

Proposed Tests

Test 1: LLM outputs duplicate of unresolved issue (primary bug scenario)

• Location: pdd/agentic_bug_orchestrator.py:441
• Input:
  • Step 1 LLM output: "Duplicate of #520"
  • Original issue pdd-change should support documentation-only changes without requiring dev units #520 state: OPEN (unresolved)
• Expected: Orchestrator validates pdd-change should support documentation-only changes without requiring dev units #520 is OPEN, logs a warning, does NOT trigger hard stop, workflow continues to Step 2
• Actual (before fix): Hard stop triggered immediately, issue closed as duplicate even though pdd-change should support documentation-only changes without requiring dev units #520 is unresolved

Test 2: LLM outputs duplicate of resolved issue (regression test)

• Location: pdd/agentic_bug_orchestrator.py:441
• Input:
  • Step 1 LLM output: "Duplicate of #520"
  • Original issue pdd-change should support documentation-only changes without requiring dev units #520 state: CLOSED (resolved)
• Expected: Hard stop triggered, workflow stops at Step 1, issue closed as duplicate
• Actual: Should continue to work as it does now (valid duplicate detection)

Test 3: LLM outputs duplicate with merged PR reference

• Location: pdd/agentic_bug_orchestrator.py:441
• Input:
  • Step 1 LLM output: "Duplicate of #520 (resolved in PR #525)"
  • Original issue pdd-change should support documentation-only changes without requiring dev units #520 state: CLOSED
• Expected: Hard stop triggered (original is resolved)
• Actual: Tests that the duplicate extraction regex works with additional context

Test 4: Validation handles invalid issue numbers gracefully

• Location: pdd/agentic_bug_orchestrator.py:441
• Input:
  • Step 1 LLM output: "Duplicate of #99999" (non-existent issue)
  • gh issue view returns error
• Expected: Log error, treat as unresolved (fail-safe), workflow continues
• Actual: Should not crash the orchestrator

Test 5: Validation handles network/gh CLI errors gracefully

• Location: pdd/agentic_bug_orchestrator.py:441
• Input:
  • Step 1 LLM output: "Duplicate of #520"
  • gh issue view times out or returns error
• Expected: Log error, treat as unresolved (fail-safe), workflow continues
• Actual: Should not crash the orchestrator

Test Location

• File: tests/test_issue_533_duplicate_validation.py (new file)
• Framework: pytest
• Rationale: Create a new test file specifically for issue pdd bug agents still closes duplicated issues that are not resolved #533 to:
  1. Keep the fix clearly documented with dedicated regression tests
  2. Avoid cluttering test_issue_469_duplicate_unresolved.py which focuses on prompt-level fixes
  3. Follow the project's pattern of creating issue-specific test files (e.g., test_issue_469_duplicate_unresolved.py)

Test Structure

The test file will include:

1. Unit tests for the new validation function (extracted to a helper):

   • _validate_duplicate_issue(issue_number, repo_owner, repo_name) → returns (is_resolved: bool, error: Optional[str])
   • Mock gh issue view to return various states (OPEN, CLOSED, error, timeout)

2. Integration tests for the orchestrator:

   • Mock run_agentic_task to return duplicate outputs
   • Mock subprocess.run (for gh issue view) to return issue states
   • Verify hard stop is triggered only when original is resolved
   • Verify workflow continues when original is unresolved

3. Regression tests from the live incident:

   • Simulate exact scenario from issue pdd-change finished step 1-5 but failed at step 6 #530/pdd-change should support documentation-only changes without requiring dev units #520
   • Verify fix prevents incorrect closure

Notes

• Tests will mock external dependencies (gh CLI, run_agentic_task) to ensure deterministic behavior
• The validation logic should be fail-safe: if gh issue view fails for any reason (network error, rate limit, invalid issue), treat the duplicate as unresolved and let the workflow continue
• The fix adds validation at the orchestrator level, which is the correct defense-in-depth approach:
  1. Prompt instructs LLM to check (first line of defense)
  2. Orchestrator validates LLM output (second line of defense - this is what's missing)
• Performance consideration: gh issue view is a quick operation (~100ms), acceptable overhead for Step 1
• The validation should log detailed diagnostic information to help debug future issues

---

Proceeding to Step 7: Generate Test

[prompt-driven-github]

Step 7: Generated Test

Test File

tests/test_issue_533_duplicate_validation.py

Test Code

"""
Tests for Issue #533: Orchestrator should validate LLM duplicate detection output.

Bug: The orchestrator at line 441 of agentic_bug_orchestrator.py blindly trusts
the LLM's duplicate detection output without validating that the original issue
is actually resolved. When the LLM fails to follow prompt instructions and outputs
"Duplicate of #520" even though #520 is OPEN, the orchestrator incorrectly triggers
a hard stop and closes the issue.

Root cause:
The hard stop at line 441 checks for the string "Duplicate of #" but doesn't
validate the original issue's state:
    if step_num == 1 and "Duplicate of #" in output:
        msg = f"Stopped at Step 1: Issue is a duplicate. {output.strip()}"
        return False, msg, total_cost, last_model_used, changed_files

Fix:
The orchestrator should validate the original issue's state using `gh issue view`
before triggering the hard stop. If the original issue is still OPEN (unresolved),
the orchestrator should log a warning and continue the workflow instead of stopping.

These tests verify:
1. When LLM outputs duplicate of UNRESOLVED issue → orchestrator validates & continues
2. When LLM outputs duplicate of RESOLVED issue → hard stop works correctly
3. Edge cases: invalid issue numbers, network errors, various output formats
4. Fail-safe behavior: errors default to letting workflow continue
"""

# Key tests include:
# - test_bug_orchestrator_llm_outputs_duplicate_of_unresolved_issue (PRIMARY)
# - test_bug_orchestrator_llm_outputs_duplicate_of_resolved_issue (REGRESSION)
# - test_bug_orchestrator_issue_533_exact_scenario (REGRESSION for #533)
# - test_bug_orchestrator_invalid_issue_number_failsafe (FAIL-SAFE)
# - test_bug_orchestrator_gh_cli_error_failsafe (FAIL-SAFE)

What This Test Verifies

The test suite verifies that the orchestrator validates duplicate detection before triggering hard stops. The primary test (test_bug_orchestrator_llm_outputs_duplicate_of_unresolved_issue) simulates the exact bug scenario:

1. Setup: LLM outputs "Duplicate of pdd-change should support documentation-only changes without requiring dev units #520" without checking resolution status
2. Context: Issue pdd-change should support documentation-only changes without requiring dev units #520 is still OPEN (unresolved)
3. Expected behavior: Orchestrator should:
   • Call gh issue view #520 to verify state
   • See that pdd-change should support documentation-only changes without requiring dev units #520 is OPEN
   • Log a warning
   • Continue workflow to Step 2+
4. Buggy behavior (before fix):
   • Line 441 sees "Duplicate of #" string
   • Immediately triggers hard stop
   • Returns success=False
   • Issue gets closed as duplicate

The test correctly FAILS on the current buggy code with:

AssertionError: Workflow should continue when LLM outputs duplicate of UNRESOLVED issue. 
The orchestrator should validate that #520 is OPEN and not trigger hard stop. 
Instead got: success=False, msg=Stopped at Step 1: Issue is a duplicate.

This demonstrates that the orchestrator is blindly trusting the LLM output without validation, which is the root cause identified in Step 5.

Running the Test

# Run the primary bug detection test
pytest tests/test_issue_533_duplicate_validation.py::test_bug_orchestrator_llm_outputs_duplicate_of_unresolved_issue -v

# Run all tests for issue #533
pytest tests/test_issue_533_duplicate_validation.py -v

# Expected: 8 failures, 2 passes
# - Failures: Tests expecting validation (not yet implemented)
# - Passes: Regression test for resolved duplicates, no duplicate marker test

---

Proceeding to Step 8: Verification

[prompt-driven-github]

Step 8: Verification Complete

Test Execution

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.3.5, pluggy-1.6.0
collected 10 items

tests/test_issue_533_duplicate_validation.py::test_bug_orchestrator_llm_outputs_duplicate_of_unresolved_issue FAILED [ 10%]
tests/test_issue_533_duplicate_validation.py::test_bug_orchestrator_llm_outputs_duplicate_of_resolved_issue PASSED [ 20%]
tests/test_issue_533_duplicate_validation.py::TestDuplicateOutputFormats::test_duplicate_extraction_formats[standard_format] FAILED [ 30%]
tests/test_issue_533_duplicate_validation.py::TestDuplicateOutputFormats::test_duplicate_extraction_formats[with_pr_reference] FAILED [ 40%]
tests/test_issue_533_duplicate_validation.py::TestDuplicateOutputFormats::test_duplicate_extraction_formats[multiple_issues] FAILED [ 50%]
tests/test_issue_533_duplicate_validation.py::TestDuplicateOutputFormats::test_duplicate_extraction_formats[with_markdown] FAILED [ 60%]
tests/test_issue_533_duplicate_validation.py::test_bug_orchestrator_invalid_issue_number_failsafe FAILED [ 70%]
tests/test_issue_533_duplicate_validation.py::test_bug_orchestrator_gh_cli_error_failsafe FAILED [ 80%]
tests/test_issue_533_duplicate_validation.py::test_bug_orchestrator_issue_533_exact_scenario FAILED [ 90%]
tests/test_issue_533_duplicate_validation.py::test_bug_orchestrator_no_validation_without_duplicate_marker PASSED [100%]

=========================== 8 failed, 2 passed in 3.91s ==========================

Key Failure Message:
AssertionError: Workflow should continue when LLM outputs duplicate of UNRESOLVED issue. 
The orchestrator should validate that #520 is OPEN and not trigger hard stop. 
Instead got: success=False, msg=Stopped at Step 1: Issue is a duplicate.

Verification Status

✅ PASS: Test correctly detects the bug

The test suite successfully detects the bug reported in issue #533. 8 out of 10 tests fail with the expected error, demonstrating that the orchestrator incorrectly triggers a hard stop when the LLM outputs "Duplicate of #" without validating that the original issue is actually resolved.

The 2 passing tests are regression tests that verify:

1. Valid duplicate detection (when original is CLOSED) still works correctly
2. No false positives when output doesn't contain "Duplicate of #" marker

Summary

Step	Result
Duplicate Check	✅ Not a duplicate
Documentation	✅ Legitimate bug confirmed
Triage	✅ Sufficient context to proceed
Reproduction	✅ Bug reproduced with issue #530/#520
Root Cause	✅ Identified at line 441
Prompt Classification	✅ Code bug, not prompt defect
Test Plan	✅ Comprehensive test plan created
Test Generation	✅ Test suite generated
Verification	✅ Tests correctly detect the bug

Bug Details

• Location: pdd/agentic_bug_orchestrator.py:441
• Root Cause: Orchestrator blindly trusts LLM duplicate detection without validating that the original issue is resolved
• Test File: tests/test_issue_533_duplicate_validation.py

Technical Analysis

The bug is a regression of issue #469. While the prompts were fixed to instruct the LLM to check resolution status, there is no orchestrator-level validation as a defense-in-depth measure.

Current buggy code (line 441):

if step_num == 1 and "Duplicate of #" in output:
    msg = f"Stopped at Step 1: Issue is a duplicate. {output.strip()}"
    return False, msg, total_cost, last_model_used, changed_files

Problem: The code simply checks for the string "Duplicate of #" and immediately stops the workflow without verifying the original issue's state.

Required fix: Add validation logic that:

1. Extracts the duplicate issue number from the LLM output
2. Calls gh issue view to check if the original issue is actually CLOSED
3. Only triggers the hard stop if the original issue is resolved
4. Logs a warning and continues the workflow if the original is still OPEN
5. Implements fail-safe behavior (continue on errors)

Test Coverage

The test suite includes 10 comprehensive tests:

1. Primary bug scenario: LLM outputs duplicate of unresolved issue → should continue
2. Regression test: LLM outputs duplicate of resolved issue → hard stop should work
3. Output format variations: 4 tests for different duplicate declaration formats
4. Fail-safe tests: Invalid issue numbers, network errors, CLI timeouts
5. Exact scenario from pdd bug agents still closes duplicated issues that are not resolved #533: Regression test using actual issue numbers pdd-change finished step 1-5 but failed at step 6 #530/pdd-change should support documentation-only changes without requiring dev units #520
6. No false positives: Conversational mentions shouldn't trigger validation

Next Steps

1. Fix the bug at pdd/agentic_bug_orchestrator.py:441 by adding validation logic
2. Run the test suite to confirm the fix: pytest tests/test_issue_533_duplicate_validation.py
3. Run the full test suite to check for regressions: pytest tests/
4. Submit PR with the fix and test

---

Proceeding to Step 9: E2E Tests

[prompt-driven-github]

Step 9: E2E Test

Environment Discovery

• Repository language: Python
• Test framework: pytest
• Existing E2E tests found: Yes - Multiple E2E tests found in tests/ directory (e.g., test_e2e_issue_469_duplicate_unresolved.py, test_e2e_issue_305_false_success.py, test_agentic_e2e_fix_orchestrator.py)
• Test configuration: pytest.ini with custom markers including e2e: mark a test as E2E test
• Fixtures/helpers used:
  • set_pdd_path_to_project_root - Ensures PDD_PATH points to project root for real prompt loading
  • mock_git_repo - Creates minimal git repository for testing
  • Real prompt loading via load_prompt_template (not mocked)
  • Real preprocessing via preprocess() function
• Environment variables required: PDD_PATH (set to project root), PDD_RUN_ALL_TESTS
• Auth/credentials: Tests use mocked GitHub CLI (gh) responses and mocked LLM calls

E2E Test File

tests/test_e2e_issue_533_duplicate_validation.py

Test Type

Integration

What This Test Verifies

This E2E test verifies the user-facing behavior of the agentic bug workflow when the LLM fails to follow prompt instructions and outputs "Duplicate of #520" without checking that issue #520 is still OPEN (unresolved).

User-facing scenario from issue #533:

1. User files issue pdd-change finished step 1-5 but failed at step 6 #530 about a bug
2. The pdd bug agent runs and LLM Step 1 outputs "Duplicate of pdd-change should support documentation-only changes without requiring dev units #520"
3. Issue pdd-change should support documentation-only changes without requiring dev units #520 is still OPEN (not resolved)
4. BUGGY: Orchestrator stops workflow and closes pdd-change finished step 1-5 but failed at step 6 #530 as duplicate
5. EXPECTED: Orchestrator validates pdd-change should support documentation-only changes without requiring dev units #520 is OPEN and continues workflow

The E2E test exercises the full pipeline:

• Real prompt loading from disk
• Real preprocessing and formatting
• Real orchestrator loop logic with step iteration
• Only LLM execution and git operations are mocked

Test Code

"""
E2E tests for Issue #533: Orchestrator should validate LLM duplicate detection output.

These E2E tests differ from the unit tests in test_issue_533_duplicate_validation.py:
- Unit tests mock load_prompt_template and test orchestrator logic in isolation
- E2E tests use REAL prompt loading via load_prompt_template (not mocked)
- E2E tests exercise the full pipeline: prompt file → preprocess → format → orchestrator

Bug: The orchestrator at line 441 of agentic_bug_orchestrator.py blindly trusts
the LLM's duplicate detection output without validating that the original issue
is actually resolved. When the LLM fails to follow prompt instructions and outputs
"Duplicate of #520" even though #520 is OPEN, the orchestrator incorrectly triggers
a hard stop and closes the issue.

Real-world scenario (issue #530/#520):
- User files issue #530 about a bug
- LLM outputs "Duplicate of #520" without verifying #520's status
- Issue #520 is still OPEN (unresolved)
- Orchestrator blindly trusts LLM and stops workflow, closing #530
- User had to manually reopen #530

Root cause:
The hard stop at line 441 checks for "Duplicate of #" but doesn't validate:
    if step_num == 1 and "Duplicate of #" in output:
        msg = f"Stopped at Step 1: Issue is a duplicate. {output.strip()}"
        return False, msg, total_cost, last_model_used, changed_files

Fix:
The orchestrator should validate the original issue's state using \`gh issue view\`
before triggering the hard stop. If the original issue is still OPEN (unresolved),
the orchestrator should log a warning and continue the workflow.

This is a regression of issue #469, which fixed the prompts but didn't add
orchestrator-level validation as defense-in-depth against LLM instruction-following failures.

Test Strategy:
- Test 1: LLM outputs duplicate of UNRESOLVED issue → orchestrator validates & continues
- Test 2: LLM outputs duplicate of RESOLVED issue → hard stop works correctly (regression)
- Test 3: Exact scenario from issue #533 using real issue numbers #530/#520
"""

import os
import re
import subprocess
from pathlib import Path
from unittest.mock import patch

import pytest

from pdd.agentic_bug_orchestrator import run_agentic_bug_orchestrator

# Project root: the worktree (or repo root) containing prompts/
_PROJECT_ROOT = Path(__file__).resolve().parent.parent


@pytest.fixture(autouse=True)
def set_pdd_path_to_project_root():
    """Ensure PDD_PATH points to the project root so load_prompt_template
    picks up the prompts/ directory from this worktree, not an external install."""
    old = os.environ.get("PDD_PATH")
    os.environ["PDD_PATH"] = str(_PROJECT_ROOT)
    yield
    if old is not None:
        os.environ["PDD_PATH"] = old
    elif "PDD_PATH" in os.environ:
        del os.environ["PDD_PATH"]


@pytest.fixture
def mock_git_repo(tmp_path):
    """Create a minimal git repository for testing the orchestrator."""
    repo_path = tmp_path / "test_repo"
    repo_path.mkdir()

    subprocess.run(
        ["git", "init", "-b", "main"], cwd=repo_path,
        check=True, capture_output=True
    )
    subprocess.run(
        ["git", "config", "user.email", "test@test.com"],
        cwd=repo_path, check=True
    )
    subprocess.run(
        ["git", "config", "user.name", "Test User"],
        cwd=repo_path, check=True
    )

    (repo_path / "README.md").write_text("# Test Repository\n")
    subprocess.run(["git", "add", "."], cwd=repo_path, check=True)
    subprocess.run(
        ["git", "commit", "-m", "Initial commit"],
        cwd=repo_path, check=True, capture_output=True
    )

    return repo_path


class TestIssue533DuplicateValidationE2E:
    """
    E2E tests for Issue #533: Orchestrator should validate duplicate detection.

    These tests exercise the real prompt loading, preprocessing, and formatting
    pipeline — only the LLM execution layer (run_agentic_task) and git
    operations (_setup_worktree) are mocked.
    """

    def test_llm_outputs_duplicate_of_unresolved_issue_workflow_continues(self, mock_git_repo):
        # ... (full test implementation in the file)

    def test_llm_outputs_duplicate_of_resolved_issue_workflow_stops(self, mock_git_repo):
        # ... (regression test - ensures valid duplicate detection still works)

    def test_exact_scenario_issue_533_with_real_issue_numbers(self, mock_git_repo):
        # ... (exact reproduction of issue #533 scenario with #530/#520)

Full test file: tests/test_e2e_issue_533_duplicate_validation.py

Test Execution Result

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.3.5, pluggy-1.6.0
collected 3 items

tests/test_e2e_issue_533_duplicate_validation.py::TestIssue533DuplicateValidationE2E::test_llm_outputs_duplicate_of_unresolved_issue_workflow_continues FAILED [ 33%]
tests/test_e2e_issue_533_duplicate_validation.py::TestIssue533DuplicateValidationE2E::test_llm_outputs_duplicate_of_resolved_issue_workflow_stops PASSED [ 66%]
tests/test_e2e_issue_533_duplicate_validation.py::TestIssue533DuplicateValidationE2E::test_exact_scenario_issue_533_with_real_issue_numbers FAILED [100%]

=================================== FAILURES ===================================
_ TestIssue533DuplicateValidationE2E.test_llm_outputs_duplicate_of_unresolved_issue_workflow_continues _

AssertionError: BUG DETECTED (Issue #533): Workflow should continue when LLM outputs 
duplicate of UNRESOLVED issue. The orchestrator should validate that #520 is OPEN and 
not trigger hard stop. Instead got: success=False, msg=Stopped at Step 1: Issue is a duplicate.

_ TestIssue533DuplicateValidationE2E.test_exact_scenario_issue_533_with_real_issue_numbers _

AssertionError: BUG DETECTED (Issue #533 - Exact Scenario): This reproduces the exact 
bug reported in issue #533 where issue #530 was incorrectly closed as a duplicate of 
the still-OPEN issue #520. The orchestrator should validate that #520 is OPEN and 
continue the workflow. Instead got: success=False, msg=Stopped at Step 1: Issue is a duplicate.

========================= 2 failed, 1 passed in 3.69s ==========================

E2E Test Status

PASS: E2E test correctly detects the bug

The E2E test suite successfully detects the bug at line 441 of pdd/agentic_bug_orchestrator.py:

Test Results:

• ❌ test_llm_outputs_duplicate_of_unresolved_issue_workflow_continues - FAILED (detects the bug)

  • When LLM outputs "Duplicate of pdd-change should support documentation-only changes without requiring dev units #520" but pdd-change should support documentation-only changes without requiring dev units #520 is OPEN, orchestrator should validate and continue
  • Currently orchestrator blindly stops the workflow (buggy behavior)

• ✅ test_llm_outputs_duplicate_of_resolved_issue_workflow_stops - PASSED (regression test)

  • When LLM correctly identifies a CLOSED duplicate, workflow stops correctly
  • This ensures the fix won't break valid duplicate detection

• ❌ test_exact_scenario_issue_533_with_real_issue_numbers - FAILED (reproduces exact bug)

  • Uses real issue numbers pdd-change finished step 1-5 but failed at step 6 #530/pdd-change should support documentation-only changes without requiring dev units #520 from the bug report
  • Confirms the exact scenario where pdd-change finished step 1-5 but failed at step 6 #530 was incorrectly closed as duplicate of OPEN issue pdd-change should support documentation-only changes without requiring dev units #520

Bug Confirmed: The orchestrator at line 441 checks for "Duplicate of #" in the LLM output and immediately triggers a hard stop without validating that the original issue is actually resolved. This causes issues to be incorrectly closed when the LLM fails to follow prompt instructions.

E2E Test Difference from Unit Tests:
The E2E test exercises the full code path including:

• Real prompt file loading from disk
• Real preprocessing (include expansion, brace escaping)
• Real formatting with context variables
• Real orchestrator loop logic with all 11 steps
• Only LLM execution (run_agentic_task) and git operations (_setup_worktree) are mocked

This provides higher confidence that the fix will work in production.

---

Proceeding to Step 10: Create Draft PR