Test AI Code Review

[diegotl]

Testing the AI code review workflow with GLM 4.7

[github-actions]

🧠 AI Code Review

Summary

This GitHub Actions workflow implements an automated code review bot. It triggers on pull requests, generates a diff, and sends it to an external AI API (configured via secrets) to generate a review summary. The review is then posted as a comment on the PR, with logic to update existing comments rather than creating duplicates.

---

Issues

1. Git Diff Reliability:
   The command git diff origin/${{ github.base_ref }}...HEAD assumes the base branch ref exists locally in the expected state. In GitHub Actions, especially with forks or complex branching strategies, this can be flaky.

   • Fix: Use git diff ${{ github.event.pull_request.base.sha }}...HEAD. This uses the specific commit SHA that GitHub identifies as the PR target, which is more robust.

2. Comment Identification Logic:
   The script identifies the bot's comment by checking comment.user.type === 'Bot' && comment.body.includes('AI Code Review').

   • Risk: If another bot (e.g., Dependabot or a security linter) leaves a comment containing the text "AI Code Review", this script will accidentally overwrite that comment.
   • Fix: Check specifically for the actor/user login (e.g., comment.user.login === 'github-actions[bot]') or a unique, random marker in the comment body.

3. Byte-Level Truncation:
   Using head -c 50000 truncates the diff at an arbitrary byte boundary.

   • Risk: This could cut a line of code in half or split a multibyte Unicode character, potentially causing the AI to misinterpret the code or fail to parse the diff.

---

Security

1. Secret Leakage via Logs:
   While set +x is used, the workflow constructs the curl command line. If debug logging is enabled elsewhere or verbose modes are triggered, the Authorization: Bearer $API_KEY header could theoretically be exposed in runner logs (though GitHub attempts to mask secrets). It is generally safer to pass secrets via a file or environment variable specifically supported by the tool, but curl usage here is standard.

2. Injection Risk:
   The DIFF variable is correctly JSON-escaped using jq -Rs . before being used in the payload. This effectively prevents shell injection or JSON structure corruption from the diff content.

---

Performance

1. Context Window Usage:
   The MAX_SIZE is set to 50,000 bytes (~50KB). This is very conservative for modern LLMs (e.g., GPT-4, Claude 3), which support context windows of 100k+ tokens (hundreds of KB).
   • Impact: Large PRs will be aggressively truncated, potentially missing critical context or changes.
   • Suggestion: Increase the limit or allow it to be configured via a variable/secret.

---

Best Practices

1. Hardcoded Language Context:
   The system prompt explicitly asks to review for "Swift 6, concurrency...".

   • Observation: This hardcodes the workflow for Swift projects. If this workflow is intended to be reusable across a multi-language repository, the system prompt should be configurable via a workflow input or secret.

2. Shell Script Quoting:
   Constructing the complex JSON payload inside the curl -d argument using double quotes and variable interpolation is difficult to read and maintain.

   • Suggestion: Use jq to construct the entire request JSON object from shell variables, which handles escaping and formatting automatically.

3. Error Handling:
   The script exits with exit 1 if the API fails. This is correct behavior for a CI check, but consider adding a "continue-on-error" option or a softer failure mode if the AI service is down, so it doesn't block all merges strictly due to an API timeout.

---

Suggestions

1. Improve Diff Reliability:

   # Use the specific SHA provided by the PR event context
   git diff ${{ github.event.pull_request.base.sha }}...HEAD > pr.diff

2. Use jq for Payload Construction:
   Replace the manual JSON string construction with a jq filter for safety and readability:

   PAYLOAD=$(jq -n \
     --arg model "$MODEL" \
     --arg diff "$DIFF" \
     --arg system_prompt "You are an expert code reviewer..." \
     '{
       model: $model,
       messages: [
         {role: "system", content: $system_prompt},
         {role: "user", content: $diff}
       ],
       temperature: 0.3,
       max_tokens: 4096
     }')
   
   curl -s -o response.json -w "%{http_code}" $API_ENDPOINT \
     -H "Authorization: Bearer $API_KEY" \
     -H "Content-Type: application/json" \
     -d "$PAYLOAD"

3. Make System Prompt Configurable:
   Add an environment variable for the system prompt so the workflow can be adapted for other languages without editing the YAML.

4. Safer Comment Matching:

   const botComment = comments.find(comment =>
     comment.user.login === 'github-actions[bot]' && // Specific bot login
     comment.body.includes('AI Code Review')
   );