feat: Universal LiteLLM Backend (Issue #234)#296
feat: Universal LiteLLM Backend (Issue #234)#296whoisjayd wants to merge 30 commits intovitali87:mainfrom
Conversation
- Replace legacy Google/OpenAI/Ollama providers with generic LiteLLMProvider - Update configuration to support dynamic provider routing - Add dynamic provider listing in constants - Remove hardcoded Google GenAI dependency from DocumentAnalyzer - Update dependencies to include litellm>=1.52.0
- Add test_litellm_provider_full.py for comprehensive provider testing - Update test_tool_calling.py to use pydantic_ai.models.test.TestModel - Fix type checks in conftest.py - Update test_provider_classes.py for dynamic provider list and remove legacy providers - Update test_document_analyzer.py to mock LiteLLM provider logic
Includes detailed setup for all major providers, troubleshooting, and best practices.
Generated dynamic list of 100+ supported providers and updated main documentation.
Summary of ChangesHello @whoisjayd, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a significant architectural overhaul to the LLM integration, transitioning from a fragmented, provider-specific setup to a unified, highly extensible LiteLLM backend. This change dramatically expands the range of supported language models and enables advanced features like custom API headers for gateway integration, all while simplifying the underlying codebase and improving future maintainability. The core purpose is to provide a flexible and future-proof foundation for interacting with diverse LLM services. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This is an excellent and significant refactoring that replaces multiple provider-specific implementations with a universal LiteLLMProvider. This change greatly enhances the tool's flexibility, supporting over 100 LLM providers without code modifications. The codebase is now cleaner, more maintainable, and the new documentation is a valuable addition. I've identified a few areas for improvement, including a potential race condition when handling API headers, some dead code that can be removed for clarity, and a minor regression in error handling specificity in one of the tools, which has been updated to reference repository guidelines. Overall, this is a fantastic contribution that significantly improves the project's architecture.
Greptile OverviewGreptile SummaryThis PR successfully replaces the legacy multi-provider architecture with a unified LiteLLM backend, enabling support for 100+ LLM providers through a single implementation. The refactor removes ~176k lines (mostly docs/optimize artifacts) and adds comprehensive test coverage. Major Changes:
Issues Found:
The architecture is sound and the implementation correctly uses PydanticAI's model abstraction, but the test bugs need immediate fixing before merge. Confidence Score: 3/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant Main
participant Config
participant BaseProvider
participant LiteLLMProvider
participant PydanticLiteLLM
participant OpenAIChatModel
participant Agent
User->>Main: Start application
Main->>Config: Load settings from .env
Config-->>Main: ModelConfig (provider, model_id, api_key, etc.)
Main->>BaseProvider: get_provider_from_config(config)
BaseProvider->>BaseProvider: Check PROVIDER_REGISTRY
BaseProvider->>LiteLLMProvider: new LiteLLMProvider(provider, **config)
LiteLLMProvider-->>BaseProvider: provider instance
BaseProvider-->>Main: provider instance
Main->>LiteLLMProvider: create_model(model_id)
LiteLLMProvider->>LiteLLMProvider: validate_config()
LiteLLMProvider->>LiteLLMProvider: Build full_model_id (provider/model)
LiteLLMProvider->>PydanticLiteLLM: new PydanticLiteLLMProvider(api_key, api_base)
alt Vertex AI provider
LiteLLMProvider->>LiteLLMProvider: Enter _vertex_env_context
Note over LiteLLMProvider: Set VERTEXAI_PROJECT, VERTEXAI_LOCATION env vars
end
LiteLLMProvider->>LiteLLMProvider: Build ModelSettings (extra_headers, extra_body, timeout)
LiteLLMProvider->>OpenAIChatModel: new OpenAIChatModel(full_model_id, provider, settings)
alt Error handling
OpenAIChatModel-->>LiteLLMProvider: RateLimitError/Timeout/InvalidRequestError
LiteLLMProvider-->>Main: Wrapped ValueError with context
end
OpenAIChatModel-->>LiteLLMProvider: model instance
alt Vertex AI provider
LiteLLMProvider->>LiteLLMProvider: Exit _vertex_env_context
Note over LiteLLMProvider: Restore original env vars
end
LiteLLMProvider-->>Main: model instance
Main->>Agent: new Agent(model, tools, system_prompt)
Agent-->>Main: agent instance
Main->>Agent: run_sync(user_query)
Agent->>OpenAIChatModel: Generate response via LiteLLM
OpenAIChatModel->>PydanticLiteLLM: API call to provider
PydanticLiteLLM-->>OpenAIChatModel: LLM response
OpenAIChatModel-->>Agent: Parsed response
Agent-->>Main: RunResult with output
Main-->>User: Display result
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR modernizes the LLM provider architecture by replacing legacy provider-specific implementations (Google, OpenAI, Ollama) with a unified LiteLLMProvider backend. This enables support for 100+ LLM providers through LiteLLM's abstraction layer, including OpenAI, Anthropic, Google Gemini, DeepSeek, Ollama, Azure, Groq, and more. The PR also introduces support for custom headers to enable integration with third-party API gateways like Portkey and Helicone.
Changes:
- Implemented universal
LiteLLMProviderreplacing three legacy provider classes - Added dynamic provider discovery from LiteLLM's provider list via
get_all_providers() - Refactored
DocumentAnalyzerto use unified backend instead of Google-specific SDK - Added comprehensive documentation for LLM configuration and supported providers
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Added litellm 1.81.7 and its dependency fastuuid 0.14.0 |
pyproject.toml |
Added litellm>=1.52.0 dependency |
codebase_rag/providers/litellm.py |
New universal provider implementation using LiteLLM |
codebase_rag/providers/interface.py |
Extracted base ModelProvider interface |
codebase_rag/providers/base.py |
Refactored to use LiteLLMProvider for all providers |
codebase_rag/constants.py |
Added dynamic provider discovery function |
codebase_rag/config.py |
Added extra_headers support and updated model string parsing |
codebase_rag/tools/document_analyzer.py |
Migrated from google-generativeai SDK to unified backend |
codebase_rag/services/llm.py |
Updated local model detection logic |
codebase_rag/main.py |
Simplified endpoint configuration display |
codebase_rag/tests/test_provider_classes.py |
Updated tests for LiteLLMProvider |
codebase_rag/tests/test_litellm_provider_full.py |
New comprehensive tests for LiteLLM integration |
codebase_rag/tests/test_document_analyzer.py |
Updated to test with mocked Agent instead of Google SDK |
docs/LLM_CONFIGURATION.md |
New comprehensive configuration guide |
docs/SUPPORTED_PROVIDERS.md |
Auto-generated list of 2800+ supported model strings |
README.md |
Added links to new documentation |
Comments suppressed due to low confidence (1)
codebase_rag/config.py:323
- The
parse_model_string()method now prioritizes "/" over ":" as a delimiter (checking "/" first at line 315). This could be a breaking change if users have model strings containing "/" that were previously parsed differently. The logic should handle edge cases where both separators are present in the string.
def parse_model_string(self, model_string: str) -> tuple[str, str]:
if "/" in model_string:
provider, model = model_string.split("/", 1)
return provider.lower(), model
if ":" not in model_string:
return cs.Provider.OLLAMA, model_string
provider, model = model_string.split(":", 1)
if not provider:
raise ValueError(ex.PROVIDER_EMPTY)
return provider.lower(), model
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
/gemini review |
|
/gemini summary |
There was a problem hiding this comment.
Code Review
This is a significant and well-executed pull request that refactors the LLM provider architecture to use LiteLLM, greatly improving provider support and maintainability. The removal of legacy provider classes and the introduction of a universal LiteLLMProvider is a major step forward. The new documentation is also very thorough and helpful. I've identified a few areas for improvement to enhance robustness and correctness, particularly around configuration validation and provider instantiation logic, all of which align with best practices for error handling and code clarity.
Summary of ChangesThis pull request significantly overhauls the system's interaction with Large Language Models by migrating from a legacy multi-provider setup to a unified LiteLLM backend. This strategic shift dramatically expands compatibility with a vast array of LLM services and introduces flexible configuration options, such as custom request headers for API gateways. The change aims to streamline future integrations, enhance maintainability, and provide a robust, adaptable foundation for all LLM-related functionalities. Highlights
Changelog
Activity
|
|
/gemini review |
|
@greptile |
There was a problem hiding this comment.
Code Review
This is an excellent pull request that significantly refactors the LLM provider architecture to use litellm as a universal backend. This change greatly increases the number of supported LLM providers and simplifies the configuration and codebase by removing provider-specific classes. The introduction of extra_headers support is also a valuable addition for integrating with API gateways. The code changes are well-structured, and the new documentation is comprehensive and very helpful for users. I've found a couple of minor issues in the new tests, but overall, this is a fantastic improvement for the project.
codebase_rag/tests/integration/test_document_analyzer_integration.py
Outdated
Show resolved
Hide resolved
… and implementation
|
/gemini review |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request is an excellent and significant refactoring that unifies the LLM backend by introducing a universal LiteLLMProvider. This greatly enhances the project's flexibility, adding support for over 100 LLM providers and enabling integration with API gateways through custom headers. The legacy provider-specific classes have been cleanly removed, and the configuration has been centralized and improved. The addition of comprehensive documentation and a thorough test suite, including concurrency tests, demonstrates high-quality engineering.
My review includes a suggestion to complete an incomplete concurrency test and a recommendation to add back some core system settings to the .env.example file for user clarity. Overall, this is a fantastic contribution that significantly improves the project's architecture and usability.
… header test assertions
|
/gemini review |
There was a problem hiding this comment.
Code Review
This is an excellent and significant refactoring that replaces the specific provider implementations with a universal LiteLLM backend. This greatly expands model compatibility, simplifies configuration, and reduces maintenance. The introduction of support for extra headers for API gateways is a great addition. The code is well-structured, and the inclusion of comprehensive tests, especially for concurrency, is commendable. I have found one potential issue with model string construction that could affect models with colons in their names, but otherwise, this is a very solid pull request.
This comment was marked as resolved.
This comment was marked as resolved.
|
🤷 |
|
This is i guess my last attempt 😶🌫️ |
|
/gemini review |
|
@greptile |
|
/gemini summary |
There was a problem hiding this comment.
Code Review
This pull request is a major and well-executed refactoring to replace the multi-provider LLM architecture with a universal backend using LiteLLM. This is a significant improvement that enables support for over 100 providers. The changes are extensive, touching configuration, provider implementation, documentation, and tests. The new LiteLLMProvider is robust, with considerations for thread safety and detailed error handling. The documentation has been significantly expanded and improved, which will be great for users. The test suite has also been updated and expanded with new tests for concurrency and the new provider, which is excellent. Overall, this is a high-quality contribution. I have one suggestion for improvement regarding header validation, as detailed in the comment.
Summary of ChangesThis pull request introduces a significant architectural overhaul to the system's Large Language Model (LLM) integration. By transitioning from a fragmented, provider-specific setup to a unified LiteLLM backend, the project dramatically expands its compatibility with a vast array of LLM services. This change not only simplifies the codebase and enhances maintainability but also provides a flexible and future-proof foundation for interacting with diverse LLM providers and advanced features like custom API gateways. Highlights
Changelog
Ignored Files
Activity
|
|
@gemini-code-assist @greptile
Keep as-is or create Pydantic wrapper? |
|
Thank you for the clarification, @whoisjayd. You are absolutely correct that Given that |
|
You're correct. Looking at your test code (lines 123, 36, etc.), your current implementation using Recommendation: Keep as-is The TypedDict approach is appropriate here since:
No changes needed - your implementation matches PydanticAI's design. |
@vitali87 The failing test appears to be caused by a transient network issue rather than a code regression, so I’ve left it unchanged for now. I believe it can be mitigated with the cache handling fix shown above. Please review the PR when you have a chance Also i have noticed you haven't set codecov token in secrets it would be better to store it : https://app.codecov.io/gh/vitali87/code-graph-rag |
1 participant
Summary
This PR replaces the legacy multi-provider architecture (Google, OpenAI, Ollama specific classes) with a single
LiteLLMProvider. This enables support for 100+ LLM providers including OpenAI, Anthropic, Gemini, DeepSeek, Ollama, Azure, and more—all without code changes.It also introduces support for extra headers, enabling integration with 3rd party gateways like Portkey and Helicone.
Key Changes
codebase_rag/providers/litellm.py.GoogleProvider,OpenAIProvider, andOllamaProvider.constants.pynow dynamically lists available providers from the installedlitellmversion.DocumentAnalyzertool no longer depends ongoogle-generativeaiSDK; it uses the unified backend.docs/SUPPORTED_PROVIDERS.mdanddocs/LLM_CONFIGURATION.md.Issues Resolved
Test Plan
make testpassed (3000+ tests).test_litellm_provider_full.pycovering dynamic loading, config mapping, extra headers, and error handling.DocumentAnalyzerworks with mocked backend responses.