Skip to content

feat: add LiteLLM as LLM provider#833

Open
RheagalFire wants to merge 2 commits into
eracle:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as LLM provider#833
RheagalFire wants to merge 2 commits into
eracle:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

[## Summary

Adds LiteLLM as a new LLM provider option, enabling access to 100+ LLM backends (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Ollama, etc.) through a single LiteLLM proxy server.

LiteLLM appears as a selectable "LiteLLM" option in the Django admin provider dropdown alongside the existing providers.

How it works

LiteLLM runs as an OpenAI-compatible proxy, so the implementation reuses pydantic-ai's existing OpenAIModel/OpenAIProvider with a custom base_url pointing at the LiteLLM proxy. No new dependencies are required.

Configuration (Django admin SiteConfig):

  • LLM Provider: select "LiteLLM"
  • LLM API Base: your LiteLLM proxy URL (e.g. http://localhost:4000/v1)
  • AI Model: any model your proxy serves (e.g. anthropic/claude-sonnet-4-20250514)
  • LLM API Key: the proxy's virtual key (optional -- local proxies typically run without auth)

Changes

File What
linkedin/llm.py _build_litellm() builder + registration in _PROVIDER_BUILDERS + API key validation skip for litellm
linkedin/models.py LITELLM = "litellm", "LiteLLM" added to SiteConfig.LLMProvider choices
linkedin/migrations/0008_... Django migration adding the new choice to the llm_provider field
tests/test_litellm_provider.py 11 unit tests covering builder, validation, edge cases

Testing

Unit tests (11 passing):

  • Provider choice exists with correct label
  • Builder creates OpenAIModel with correct model name
  • llm_api_base is required (raises ValueError if empty)
  • API key is optional for litellm (local proxy without auth)
  • API key is forwarded when set (hosted proxy with auth)
  • _validated_site_config() skips API key check for litellm
  • _validated_site_config() still requires ai_model
  • Other providers (openai) still require API key
  • End-to-end get_llm_model() works for both local and hosted scenarios

Full test suite: 240 passed, 0 failures.

Manual testing against LiteLLM proxy (http://localhost:4000/v1) routing to Anthropic:

# Model discovery
curl http://localhost:4000/v1/models
# Returns: litellm/anthropic/claude-sonnet-4-20250514

# Completion through proxy
Provider: litellm
Model: anthropic/claude-sonnet-4-20250514
Result: successful completion with correct token counts

The builder follows the same AsyncOpenAI + OpenAIModel + OpenAIProvider pattern used by the existing openai_compatible provider, so streaming, tool calling, and structured output all work through the proxy.
](http://localhost:8000/admin/)

@RheagalFire
Copy link
Copy Markdown
Author

@eracle

@RheagalFire
Copy link
Copy Markdown
Author

@eracle do you have any update on this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant