Skip to content

Conversation

@devin-ai-integration
Copy link
Contributor

📥 Pull Request

📘 Description

Fixes 500 Internal Server Error on /metrics endpoint caused by missing ClickHouse functions calculate_prompt_cost and calculate_completion_cost. Since ClickHouse doesn't support user-defined functions via CREATE FUNCTION, this PR replaces the function calls with inline multiIf expressions that calculate costs directly in the SQL queries.

Key Changes:

  • Replaced calculate_prompt_cost() and calculate_completion_cost() function calls with inline multiIf expressions in metrics.py and traces.py
  • Added pricing for major LLM providers: OpenAI (GPT-4, GPT-4o, GPT-3.5), Anthropic (Claude-3), Google (Gemini), and Meta (Llama)
  • Includes fallback logic for unknown models (returns 0.0)
  • Removed unsupported CREATE FUNCTION statements from migration file

🧪 Testing

  • Verified ClickHouse multiIf syntax works correctly with direct query testing
  • Confirmed /metrics endpoint no longer returns ClickHouse function errors (now returns authentication errors, indicating the function issue is resolved)
  • Ran end-to-end test with AgentOps tracing to ensure overall system functionality remains intact
  • Generated successful trace: http://localhost:3000/sessions?trace_id=0a8f158f78c315a6bd5ac15e29a977bc

⚠️ Review Notes

  • Pricing Accuracy: Please verify the hardcoded pricing rates against current LLM provider pricing
  • Code Duplication: Cost calculation logic is now duplicated across metrics.py and traces.py - consider centralizing in the future
  • Unrelated Changes: This PR includes formatting changes in test files that should ideally be in a separate commit
  • Maintenance: Adding new models or updating pricing will now require changes in multiple locations

Link to Devin session: https://app.devin.ai/sessions/c94d96bbaff245c189372484a6eeb193
Requested by: @areibman (Alex)

areibman and others added 3 commits September 2, 2025 23:44
- Replace calculate_prompt_cost and calculate_completion_cost function calls with multiIf expressions
- Inline cost calculations directly in SQL queries for metrics.py and traces.py
- Remove unsupported CREATE FUNCTION statements from migration file
- Include pricing for major LLM providers (OpenAI, Anthropic, Google, Meta)
- Handle unknown models gracefully with 0.0 fallback
- Fixes 500 errors on /metrics endpoint due to missing functions

Co-Authored-By: Alex <meta.alex.r@gmail.com>
@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@devin-ai-integration
Copy link
Contributor Author

Closing PR as requested - checking updated main branch for new ClickHouse migrations that include cost calculation functions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants