LLM Router Server is a high-performance routing service designed for multi-model deployment scenarios, used to uniformly manage and orchestrate multiple local Large Language Model (LLM) services, Embedding models, Re-ranking models, and other inference services.
asyncio multi-model inference-server load-balancing model-serving fastapi llm llm-serving vllm llm-router ai-infrastructure llm-router-server
-
Updated
Mar 28, 2026 - Python