Skip to content

Fail fast on non-retriable LiteLLM status codes#1182

Open
yangbaechu wants to merge 1 commit intohuggingface:mainfrom
yangbaechu:fix/litellm-fail-fast-non-retriable-errors
Open

Fail fast on non-retriable LiteLLM status codes#1182
yangbaechu wants to merge 1 commit intohuggingface:mainfrom
yangbaechu:fix/litellm-fail-fast-non-retriable-errors

Conversation

@yangbaechu
Copy link

Summary

Fail fast on non-retriable LiteLLM errors instead of retrying them with exponential backoff. This keeps the existing retry behavior for transient failures, while surfacing clear configuration and authorization issues earlier.

Problem

LiteLLMClient.__call_api() currently retries almost all exceptions up to api_max_retry, which means clearly non-retriable failures can spend a long time in exponential backoff before the run fails.

Changes

  • treat LiteLLM errors with status codes 401, 403, and 404 as non-retriable
  • raise those errors immediately instead of retrying them
  • keep the existing retry loop for other errors
  • add tests verifying that:
    • retryable errors still retry
    • non-retriable status-code errors fail fast
  • skip the LiteLLM-specific tests when the optional litellm dependency is not installed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant