Have a rollback strategy by tagging AI-generated comments

[rdner]

Every AI-generated comment should have a machine-readable tag that includes the source automation including its version (commit hash? release tag?).

In case if something goes wrong with the new version of the AI-action and it spams our PRs or issues, we should have a way (script?) that removes all the comments from this malfunctioning AI-action at a certain version or in a given data range.

Currently, we have no rollback for a bad change. If an action spams PRs, we have to manually delete these comments.

[v1v]

/ai propose a plan

[github-actions]

Recommendation: implement rollback in two parts — (1) a stable machine-readable tag on every AI comment, then (2) a manual workflow_dispatch cleanup workflow that deletes comments by tag/version/date in dry-run mode first.

Evidence from current code:

• Footer is already appended globally via safe-outputs.messages.footer in .github/workflows/gh-aw-fragments/messages-footer.md:4.
• Most workflows import that fragment (e.g. .github/workflows/gh-aw-mention-in-issue.md:12, .github/workflows/gh-aw-plan.md:11), so one footer change propagates broadly.
• messages-footer is already an input (e.g. .github/workflows/gh-aw-mention-in-issue.md:46-50), so overrides are possible.
• We already have a PR-only mitigation (hide-older-comments: true) at .github/workflows/gh-aw-fragments/safe-output-add-comment-pr-hide-older.md:8, but no repo-wide rollback tooling.

Proposed implementation plan

1. Define metadata schema (v1)

• Add a single-line marker in footer, e.g.:
  gh-aw-meta:v1;source=elastic/ai-github-actions;workflow=${github.workflow};workflow_ref=${github.workflow_ref};run_id=${github.run_id};run_attempt=${github.run_attempt};actor=${github.actor};ts=${github.run_started_at}
• Keep it plain text (not hidden HTML comment) to avoid sanitizer/renderer surprises and make parsing deterministic.

2. Add tag to all AI comments/reviews

• Update .github/workflows/gh-aw-fragments/messages-footer.md (and messages-footer-no-feedback.md) to append the marker.
• For special custom footers (e.g. .github/workflows/gh-aw-update-pr-body.md:83), append the same marker to keep parity.

3. Add rollback workflow (manual)

• New workflow: .github/workflows/gh-aw-comment-rollback.yml (manual dispatch only).
• Inputs:
  source, workflow_ref (or version/ref), since, until, surfaces (issue/pr/discussion), author (default bot), dry_run (default true), max_deletes.
• Behavior:
  list comments via GitHub API pagination, filter by marker fields, print candidate list, then delete only when dry_run=false.

4. Safety and auditability

• Require explicit confirmation input for destructive runs.
• Emit summary: scanned/deleted/skipped + first N IDs.
• Optionally write artifact CSV of matched comments.

5. Tests/docs

• Add unit tests for marker parser and filter logic (good + malformed markers).
• Add docs section: “Rollback AI comments by metadata tag” with examples.

Acceptance criteria:

• New comments include parseable gh-aw-meta:v1 marker.
• Dry-run rollback can list comments for a specific workflow_ref and date window.
• Non-dry-run rollback deletes only matching comments and outputs an auditable summary.

If this direction looks good, I can implement it in a PR as: footer tagging first, rollback workflow second (separate commit for easier review).

---

What is this? | From workflow: Trigger Mention in Issue

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[strawgate]

There already is a stable machine readable tag on each comment

<!-- gh-aw-agentic-workflow: PR Actions Detective, engine: copilot, model: gpt-5.3-codex, id: 22903764658, workflow_id: gh-aw-pr-actions-detective, run: https://github.com/elastic/beats/actions/runs/22903764658 -->
<!-- gh-aw-workflow-call-id: elastic/beats/gh-aw-pr-actions-detective -->