In Production LLM analysis is failing

[yb175]

In production the LLM Analysis failed error

[yb175]

@coderabbitai what could be possible reason of errors ?

[yb175]

@coderabbitai what could be possible reason of errors ?

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
gh issue view 57 --repo yb175/pullShark --json title,body,comments,labels,state

Length of output: 1121

---

🏁 Script executed:

#!/bin/bash
# Find LLM related files
fd -e js -e ts -e py | xargs grep -l -i "llm\|openai\|gemini\|anthropic\|analysis" 2>/dev/null | head -30

echo "---TREE---"
tree -L 3 --gitignore 2>/dev/null || find . -maxdepth 3 -not -path '*/node_modules/*' -not -path '*/.git/*' | head -60

Length of output: 2066

---

🏁 Script executed:

#!/bin/bash
echo "=== analysePr.js ==="
cat backend/controllers/llm/analysePr.js

echo "=== analysis.service.js ==="
cat backend/services/analysis.service.js

echo "=== analysis.worker.js ==="
cat backend/worker/analysis.worker.js

echo "=== analysis.queue.js ==="
cat backend/queues/analysis.queue.js

echo "=== compressllm.js ==="
cat backend/utils/compressllm.js

Length of output: 10258

---

🏁 Script executed:

#!/bin/bash
echo "=== handleWebhook.js ==="
cat backend/controllers/webhook/handleWebhook.js

echo "=== llm.js route ==="
cat backend/routes/llm.js

echo "=== bullmqConnection.js ==="
cat backend/config/bullmqConnection.js

echo "=== githubjwt.js ==="
cat backend/utils/githubjwt.js

Length of output: 4985

---

@yb175, here's a thorough breakdown of the most likely causes for the "LLM Analysis failed" error in production:

---

🔍 Possible Root Causes

1. 🟥 Render.com Free Tier Cold Starts (Most Likely)

In analysePr.js, the LLM service is called at:

fetch("https://pullshark-ai.onrender.com/api/analyze", ...)

Render's free tier spins down services after ~15 minutes of inactivity. When a cold-start happens, the service can take 30–60 seconds to respond. Since there is no timeout configured, the request hangs — and depending on the hosting environment's socket timeout, it will eventually fail.

Fix: Add a timeout and consider upgrading to a paid Render tier or keeping the service warm with a cron ping.

const response = await fetch("https://pullshark-ai.onrender.com/api/analyze", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ pr: base64Payload }),
  signal: AbortSignal.timeout(60_000), // 60 second timeout
});

---

2. 🟥 LLM_ANALYSIS_URL Environment Variable Not Set in Production

In analysis.service.js (used by the webhook flow):

const llmUrl = process.env.LLM_ANALYSIS_URL?.trim();
if (!llmUrl) {
  throw new Error("LLM_ANALYSIS_URL is not set or is empty");
}

If this env var is missing or empty in your production environment (e.g., Render dashboard), every webhook-triggered analysis will immediately throw this error. Double-check it's configured.

---

3. 🟥 GITHUB_PRIVATE_KEY Newline Formatting Issue

In githubjwt.js:

const privateKey = process.env.GITHUB_PRIVATE_KEY;
jwt.sign(payload, privateKey, { algorithm: "RS256" });

PEM private keys contain literal newlines (\n). When pasted into a hosting environment's env var field, they often get stored as the literal string \n instead of actual newline characters. This causes jwt.sign to fail with an invalid key error, which propagates as an LLM_ANALYSIS_FAILED error since generateInstallationToken is called first.

Fix:

const privateKey = process.env.GITHUB_PRIVATE_KEY?.replace(/\\n/g, '\n');

---

4. 🟨 No response.ok Check in analysePr.js

const response = await fetch("https://pullshark-ai.onrender.com/api/analyze", { ... });
const data = await response.json(); // ← no response.ok check

If the LLM service returns a 4xx/5xx error with an HTML body (e.g., Render's error page), response.json() will throw a JSON parse error. You should guard against this:

if (!response.ok) {
  const text = await response.text();
  throw new Error(`LLM service error ${response.status}: ${text.slice(0, 200)}`);
}

---

5. 🟨 BullMQ Worker Not Running in Production

The analysis.worker.js is a separate long-running process. If your production setup only starts the Express server (server.js) and the worker process is not started, all jobs get enqueued in Redis but never processed — they silently pile up and eventually get marked as FAILED.

Verify your Dockerfile / start command runs the worker alongside the server.

---

6. 🟨 Redis Connection Misconfiguration

bullmqConnection.js requires REDIS_HOST, REDIS_PORT, and REDIS_PASSWORD:

return new IORedis({
  host: process.env.REDIS_HOST,
  port: Number(process.env.REDIS_PORT),
  password: process.env.REDIS_PASSWORD,
  ...
});

If any of these are wrong or the Redis instance is unreachable from production, the worker will fail to connect and no jobs will be processed.

---

✅ Quick Checklist

Check	Where
LLM_ANALYSIS_URL is set in production env	Render/hosting dashboard
GITHUB_PRIVATE_KEY has real newlines (not \n)	Render/hosting dashboard
REDIS_HOST, REDIS_PORT, REDIS_PASSWORD are correct	Render/hosting dashboard
BullMQ worker process is running	Dockerfile / start script
LLM service (pullshark-ai.onrender.com) is not sleeping	Render logs
Add timeout + response.ok check on fetch call	analysePr.js

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch issue `#57` details
gh issue view 57 --repo yb175/pullShark --json title,body,comments,labels,state

Length of output: 12778

---

@yb175, it looks like I already provided a detailed analysis in my earlier comment on this issue! Here's a quick summary of the top causes I identified:

Priority	Cause
🟥 High	Render.com free tier cold starts — the LLM service at pullshark-ai.onrender.com spins down after inactivity, causing request timeouts
🟥 High	LLM_ANALYSIS_URL env var not set in production (Render dashboard)
🟥 High	GITHUB_PRIVATE_KEY newline formatting — literal \n vs real newlines in env var
🟨 Medium	No response.ok check in analysePr.js — JSON parse errors on non-200 responses
🟨 Medium	BullMQ worker process not running in production
🟨 Medium	Redis connection misconfiguration (REDIS_HOST, REDIS_PORT, REDIS_PASSWORD)

Please check my previous comment for detailed explanations and code fixes for each of these. Let me know if you'd like me to dive deeper into any specific cause or if you've already ruled some of these out!

[yb175]

@coderabbitai please refer to latest architecture where we omitted the use of render

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
echo "=== analysePr.js ==="
cat backend/controllers/llm/analysePr.js

echo "=== analysis.service.js ==="
cat backend/services/analysis.service.js

echo "=== analysis.worker.js ==="
cat backend/worker/analysis.worker.js

echo "=== analysis.queue.js ==="
cat backend/queues/analysis.queue.js

echo "=== bullmqConnection.js ==="
cat backend/config/bullmqConnection.js

echo "=== githubjwt.js ==="
cat backend/utils/githubjwt.js

echo "=== compressllm.js ==="
cat backend/utils/compressllm.js

echo "=== handleWebhook.js ==="
cat backend/controllers/webhook/handleWebhook.js

echo "=== package.json ==="
cat backend/package.json

echo "=== docker/compose files ==="
fd -e yml -e yaml -e json | xargs grep -l -i "docker\|compose\|render\|railway\|fly\|gemini\|openai" 2>/dev/null | head -20

echo "=== tree ==="
tree -L 4 --gitignore 2>/dev/null || find . -maxdepth 4 -not -path '*/node_modules/*' -not -path '*/.git/*' | head -80

Length of output: 18553

---

🏁 Script executed:

#!/bin/bash
echo "=== docker-compose.yaml ==="
cat docker-compose.yaml

echo "=== backend/Dockerfile ==="
cat backend/Dockerfile

echo "=== server.js ==="
cat backend/server.js

echo "=== routes/llm.js ==="
cat backend/routes/llm.js

Length of output: 2158

---

@yb175, I've reviewed the latest architecture. The new setup uses Docker Compose with a self-hosted yb175/pullshark-llm:latest container — great move! However, I found two bugs that are likely causing the errors, plus the private key issue still applies:

---

🐛 Bug 1: analysePr.js Still Hardcodes the Render URL

The manual trigger flow (/llm/analysePr/... route) was never updated to use the new Docker-internal LLM service:

// analysePr.js — STILL USING OLD RENDER URL ❌
const response = await fetch("https://pullshark-ai.onrender.com/api/analyze", { ... });

It should use the env var just like analysis.service.js does:

// analysePr.js — FIXED ✅
const llmUrl = process.env.LLM_ANALYSIS_URL?.trim();
if (!llmUrl) throw new Error("LLM_ANALYSIS_URL is not set or is empty");

const response = await fetch(`${llmUrl}/api/analyze`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ pr: base64Payload }),
  signal: AbortSignal.timeout(60_000),
});
if (!response.ok) {
  const text = await response.text();
  throw new Error(`LLM service error ${response.status}: ${text.slice(0, 200)}`);
}

---

🐛 Bug 2: worker Service Missing LLM_ANALYSIS_URL in docker-compose.yaml

LLM_ANALYSIS_URL is only set on the backend service. The worker service only loads env_file: ./backend/.env, where this variable is likely not defined — so analysis.service.js running inside the worker throws "LLM_ANALYSIS_URL is not set or is empty":

# docker-compose.yaml — CURRENT (broken) ❌
  worker:
    build: ./backend
    command: node worker/analysis.worker.js
    env_file:
      - ./backend/.env
    restart: unless-stopped
    depends_on:
      - backend

# docker-compose.yaml — FIXED ✅
  worker:
    build: ./backend
    command: node worker/analysis.worker.js
    env_file:
      - ./backend/.env
    environment:
      - LLM_ANALYSIS_URL=http://llm:8000  # ← add this
    restart: unless-stopped
    depends_on:
      - backend
      - llm  # ← also depend on llm directly

---

⚠️ Pre-existing Issue: GITHUB_PRIVATE_KEY Newlines in githubjwt.js

This issue still applies regardless of the hosting change. If GITHUB_PRIVATE_KEY in your .env file uses literal \n instead of real newlines, jwt.sign will fail:

// githubjwt.js
const privateKey = process.env.GITHUB_PRIVATE_KEY?.replace(/\\n/g, '\n'); // ← add this

---

✅ Summary of Fixes

File	Fix
backend/controllers/llm/analysePr.js	Replace hardcoded Render URL with process.env.LLM_ANALYSIS_URL
docker-compose.yaml	Add LLM_ANALYSIS_URL=http://llm:8000 to worker service environment
backend/utils/githubjwt.js	Replace \\n with real newlines in GITHUB_PRIVATE_KEY

(_/ )
( •ᴗ•)
/ >🔍 sniffing out the bugs!

---

✏️ Learnings added

Learnt from: yb175
URL: https://github.com/yb175/pullShark/issues/57

Timestamp: 2026-03-26T08:50:35.163Z
Learning: In the pullShark project (yb175/pullShark), the LLM service is self-hosted via Docker Compose as `yb175/pullshark-llm:latest` on port 8000. The Render.com hosted LLM service (`pullshark-ai.onrender.com`) has been deprecated. `LLM_ANALYSIS_URL` should be `http://llm:8000` inside Docker Compose. The `worker` service in docker-compose.yaml needs `LLM_ANALYSIS_URL` set explicitly in its environment block (it is not inherited from the `backend` service). The `analysePr.js` controller still hardcodes the old Render URL and needs to be updated to use `process.env.LLM_ANALYSIS_URL`.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.