Summary
The Braintrust LiteLLM integration (wrap_litellm() and patch_litellm()) instruments completion, acompletion, responses, aresponses, image_generation, aimage_generation, embedding, aembedding, moderation, transcription, and atranscription. LiteLLM also exposes documented reranking APIs that are not instrumented at all:
litellm.rerank() — sync reranking for retrieval / RAG pipelineslitellm.arerank() — async reranking
LiteLLM’s rerank API is distinct from chat/completions: it takes a query plus a list of documents, and returns a RerankResponse with ranked results (index, relevance_score, optional document) plus optional metadata under meta (billed_units, tokens). Braintrust currently drops these calls entirely.
What is missing
No tracing spans are created when users call litellm.rerank() or litellm.arerank() through either wrap_litellm() or patch_litellm().
The current LiteLLM integration only patches:
completion, acompletionresponses, aresponsesimage_generation, aimage_generationembedding, aembeddingmoderationtranscription, atranscription
There are no rerank patchers in py/src/braintrust/integrations/litellm/patchers.py, no rerank wrappers in py/src/braintrust/integrations/litellm/tracing.py, and no rerank coverage in py/src/braintrust/integrations/litellm/test_litellm.py.
Braintrust docs status
not_found — The Braintrust LiteLLM integration docs at https://www.braintrust.dev/docs do not mention rerank support.
Upstream sources
- LiteLLM rerank docs: https://docs.litellm.ai/docs/rerank
- LiteLLM Python SDK signatures expose both
rerank(model, query, documents, ...) and arerank(model, query, documents, ...)
- LiteLLM
RerankResponse contains results and meta fields (litellm.types.rerank.RerankResponse)
Local repo files inspected
py/src/braintrust/integrations/litellm/__init__.pypy/src/braintrust/integrations/litellm/patchers.pypy/src/braintrust/integrations/litellm/tracing.pypy/src/braintrust/integrations/litellm/test_litellm.pypy/noxfile.py
Relationship to existing issues
Summary
The Braintrust LiteLLM integration (
wrap_litellm()andpatch_litellm()) instrumentscompletion,acompletion,responses,aresponses,image_generation,aimage_generation,embedding,aembedding,moderation,transcription, andatranscription. LiteLLM also exposes documented reranking APIs that are not instrumented at all:litellm.rerank()— sync reranking for retrieval / RAG pipelineslitellm.arerank()— async rerankingLiteLLM’s rerank API is distinct from chat/completions: it takes a
queryplus a list ofdocuments, and returns aRerankResponsewith rankedresults(index,relevance_score, optionaldocument) plus optional metadata undermeta(billed_units,tokens). Braintrust currently drops these calls entirely.What is missing
No tracing spans are created when users call
litellm.rerank()orlitellm.arerank()through eitherwrap_litellm()orpatch_litellm().The current LiteLLM integration only patches:
completion,acompletionresponses,aresponsesimage_generation,aimage_generationembedding,aembeddingmoderationtranscription,atranscriptionThere are no rerank patchers in
py/src/braintrust/integrations/litellm/patchers.py, no rerank wrappers inpy/src/braintrust/integrations/litellm/tracing.py, and no rerank coverage inpy/src/braintrust/integrations/litellm/test_litellm.py.Braintrust docs status
not_found— The Braintrust LiteLLM integration docs at https://www.braintrust.dev/docs do not mention rerank support.Upstream sources
rerank(model, query, documents, ...)andarerank(model, query, documents, ...)RerankResponsecontainsresultsandmetafields (litellm.types.rerank.RerankResponse)Local repo files inspected
py/src/braintrust/integrations/litellm/__init__.pypy/src/braintrust/integrations/litellm/patchers.pypy/src/braintrust/integrations/litellm/tracing.pypy/src/braintrust/integrations/litellm/test_litellm.pypy/noxfile.pyRelationship to existing issues
speech()/aspeech()not instrumented #165 so that issue can stay focused on the other LiteLLM API gaps.patch_litellm()does not patchembeddingormoderation;aembeddingmissing entirely #115 coverspatch_litellm()gaps for functions that already had wrapper support; this issue is specifically about rerank having no LiteLLM instrumentation at all.