feat: switch automatic PR code reviews to GPT-5.5 for model diversity

[sfreudenthaler]

Summary

Currently both automatic PR code reviews and `@claude` interactive sessions use Anthropic Claude models via AWS Bedrock. Since the team writes code with Claude's assistance, using the same model family for review can perpetuate training/weighting biases. This issue tracks switching the automatic PR review job to OpenAI GPT-5.5 via AWS Bedrock Mantle, while keeping `@claude` interactive sessions on Anthropic Claude.

Motivation

• Model diversity: A code reviewer from a different model family provides independent coverage and catches blind spots that Claude may share with the code it helped write
• Infrastructure already in place: `dotCMS/ai-workflows@v3.1.0` ships a `codex-executor` for GPT-5.5/5.4 via Bedrock Mantle; IAM policy (`IaC Dependency problem: Unable to delete contents when you have multi language version #7842`) is already applied with the `openai.*` model scope
• GPT-5.5 is entitled in us-east-1: The existing `BEDROCK_ROLE_ARN` org variable already covers the required IAM permissions — no additional credential changes needed

Proposed Changes

• `ai_claude-orchestrator.yml` — `claude-automatic-review` job:
  • Switch `model_id` from `${{ vars.BEDROCK_MODEL_ID }}` to `openai.gpt-5.5`
  • Pin to `ai-workflows@v3.1.0` (required for the GPT-5.x path fix, PR Structure, Field, ContentletAPI Unit Tests #36)
  • Update the auto-review prompt with dotCMS-specific context and structured output format
• `ai_claude-backend-reviewer.yml` — stays on Claude (uses the `Agent` tool for multi-agent sub-agents, which is Claude-specific), but update tag to `v3.1.0` and fix the Java version reference (Java 25, not Java 11)

Out of Scope

• `@claude` interactive mentions remain on `${{ vars.BEDROCK_MODEL_ID }}` (Anthropic Claude)
• `ai_claude-rollback-safety.yml` remains on Claude (label-posting logic uses Claude's tools)
• No new IAM or secrets changes needed

Acceptance Criteria

• Opening a PR triggers an automatic code review comment posted by GPT-5.5 (not Claude)
• `@claude` interactive sessions still use Anthropic Claude (verified by sticky comment header)
• Prompt in the auto-review reflects dotCMS conventions (Config, Logger, APILocator, DotConnect, WrapInTransaction)

[github-actions]

PRs linked to this issue

• feat(ai-reviews): switch automatic PR reviews to GPT-5.5 for model diversity by @sfreudenthaler
• fix(ai-review): stop models wrapping the review in a code block by @wezell

[wezell]

AI Code-Reviewer Model Analysis (dotCMS auto-review)

TL;DR

• Best value (current pick): qwen.qwen3-next-80b-a3b — caught 4/8 ground-truth bugs at ~$0.0019/review.
• Best quality: us.anthropic.claude-opus-4-8 — 5/8 bugs + 1 novel, but ~$0.081/review (~43× more) and breaks the non-Claude diversity rule.
• Drop us.deepseek.r1-v1:0 (the prior default) — 0/8.
• For the cost of ~25% more bugs, Opus is ~43× the spend. Reserve it (or an Opus+Qwen panel) for high-stakes PRs.

---

Methodology

• Input: a real dotCMS diff — the pre-hardening ImageFilterExporter SHARED_COMPLETED feature (commit e5338494bb) — plus the production gpt-auto-review.md prompt.
• Ground truth: the 8 issues a human codeowner independently found (non-atomic Files.move, nanoTime temp collision, mkdirs ignored-return, startsWith prefix bug, weak idempotent test, missing gate coverage, unbounded virtual threads/no dedup, silent-swallow on broken mount).
• Execution: each model invoked via the Bedrock path it uses in CI (Converse for *, mantle/Responses for openai.*), output parsed identically.
• Caveat: single run per model — directional, not statistically definitive. The test diff excluded the test file, so every model's "no test coverage" finding is a harness artifact (discounted equally).

---

Quality leaderboard (9 models)

Rank	Model	Real issues / 8	False positives	Latency	Open?
1	Claude Opus 4.8	5 (+1 novel)	0	fast	❌
2	Qwen3-Next-80B	4 (+novel)	~0	13s	✅
3	Qwen3-Coder-next	3–4	0	95s ⚠️	✅
4	GLM-5 (zai.glm-5)	3	0	23s	✅
5	Qwen3-Coder-30B	2–3	1	6s ⚡	✅
6	GPT-5.5 (mantle)	2 (both high-value)	0	71s ⚠️	❌
7	Mistral Pixtral Large	2	2	13s	✅
8	gpt-oss-120b	~1	2 (hallucinated)	20s	✅
9	DeepSeek R1 (prior default)	0	0	11s	✅

Notes

• Reasoning models (R1, GPT-5.5) underperformed — they spent most of their output budget on hidden reasoning and surfaced few concrete findings.
• Different model families catch different bugs: only the Qwen models caught the nanoTime collision; only Opus + GPT-5.5 caught the startsWith prefix bug. An Opus + Qwen3-Next panel would union to ~6–7/8.

---

Bedrock pricing (us-east-1, on-demand standard)

Model	Input /1M	Output /1M
Claude Opus 4.8	$5.00	$25.00
Qwen3-Next-80B-A3B	$0.14	$1.20

Qwen also offers cheaper/faster flex ($0.07 / $0.60) and pricier priority ($0.26 / $2.10) tiers.

---

Cost per review (actual A/B token usage)

Model	Tokens (in / out)	Input cost	Output cost	Per review	Per 1,000 reviews
Opus 4.8	7,100 / 1,807	$0.0355	$0.0452	$0.0807	$80.70
Qwen3-Next-80B (standard)	4,146 / 1,088	$0.00058	$0.00131	$0.0019	$1.89
Qwen3-Next-80B (flex)	4,146 / 1,088	$0.00029	$0.00065	$0.0009	$0.94

Opus 4.8 costs ~43× more per review than Qwen3-Next-80B (~85× on Qwen's flex tier). Drivers:

• Opus output is 20.8× Qwen's per-token output price ($25 vs $1.20), and output dominates review cost.
• Opus tokenizes the same prompt to ~70% more input tokens (7,100 vs 4,146).

---

Recommendation

• Default: qwen.qwen3-next-80b-a3b — ~80% of Opus's bug-catching at ~2–3% of the cost. (Currently configured.)
• High-stakes PRs: optionally an Opus + Qwen3-Next two-model panel (~6–7/8 combined) — they catch different bug classes.
• Retire: DeepSeek R1 (0/8).
• The non-Claude diversity rule (issue feat: switch automatic PR code reviews to GPT-5.5 for model diversity #36131) is the only thing keeping Opus out of the default slot; relax it only if the extra finding is worth ~43× the spend.

Infrastructure changes landed alongside this evaluation

• IAM: added qwen.* to the BedrockInvokeReviewModels policy (v4) — it previously allowed only Claude / DeepSeek / Nova / mantle.
• Prompt: #36303 stops models wrapping the whole review in a code block (Qwen was echoing the format example's fence).
• Config: REVIEW_MODEL_ID repo variable set to qwen.qwen3-next-80b-a3b; routes through the bedrock-generic Converse executor.