help request: How to represent the LLM

[liuxiaobo007]

Description

1. Call the LLM directly：
   curl https://api-inference.modelscope.cn/v1/chat/completions
   -H "Content-Type: application/json"
   -H "Authorization: Bearer xxxxxxxxxxxxxxxx"
   -d '{
   "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
   "messages": [
   {"role": "user", "content": "hello"}
   ],
   "stream": false
   }'

2. Add a route in APISIX：
   curl -X PUT http://127.0.0.1:9180/apisix/admin/routes/1
   -d '{
   "uri": "/v1/chat/completions",
   "upstream": {
   "nodes": {
   "api-inference.modelscope.cn:443": 1
   },
   "type": "roundrobin",
   "scheme": "https"
   }
   }'

3.Calling LLM through APISIX：
curl -X POST "http://127.0.0.1:9080/v1/chat/completions"
-H "Content-Type: application/json"
-H "Authorization: Bearer xxxxxxxxxxx"
-d '{
"model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
"messages": [
{"role": "user", "content": "你是谁"}
],
"stream": false
}'

I checked it again and again, but found nothing wrong. Why didn't I successfully represent the big model?
I'm using the curl-sl https://run.api7.ai/apisix/quickstart | sh command to deploy APISIX directly.

Environment

• APISIX version (run apisix version):3.15
• Operating system (run uname -a):Rocky Linux-8.10
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

Hi @liuxiaobo007, AI proxy requires specific plugins to function, you can refer to https://docs.api7.ai/hub/ai-proxy/

[liuxiaobo007]

Hi @liuxiaobo007, AI proxy requires specific plugins to function, you can refer to https://docs.api7.ai/hub/ai-proxy/

Do I have to use plug-ins? Why not directly route and forward? You can directly configure a routing forwarding with nginx to proxy the big model, but why can't APISIX?Isn't APISIX developed based on nginx?

[Baoyuantop]

Hi @liuxiaobo007, the issue is caused by the default behavior of pass_host in the APISIX upstream configuration.

When you send a request via curl http://127.0.0.1:9080/anything, the client's Host header is 127.0.0.1:9080. The default value of pass_host is "pass", which means APISIX forwards the client's original Host header as-is. This leads to:

1. APISIX sends Host: 127.0.0.1:9080 to the upstream openrouter.ai
2. The TLS SNI (Server Name Indication) during the SSL handshake is also set to 127.0.0.1:9080 (instead of openrouter.ai)
3. OpenRouter's TLS server cannot match a certificate for SNI 127.0.0.1:9080, causing the SSL handshake to fail

Add "pass_host": "node" to the upstream configuration, so APISIX uses the upstream node's domain name (openrouter.ai) as both the Host header and TLS SNI:

curl -i http://127.0.0.1:9180/apisix/admin/routes/1 \
  -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" \
  -X PUT -d '{
    "uri": "/anything",
    "upstream": {
      "type": "roundrobin",
      "pass_host": "node",
      "nodes": {
        "openrouter.ai:443": 1
      },
      "scheme": "https"
    }
  }'

Why the ai-proxy Plugin Is Recommended

While the above solution works, if your use case is proxying LLM/AI services, we strongly recommend using the ai-proxy plugin for the following reasons:

1. Automatic connection handling: ai-proxy automatically sets the correct Host, SNI, path, etc. based on the configured provider — no need to manually configure pass_host, scheme, and other low-level settings, avoiding issues like this one
2. Unified multi-model/multi-provider management: Supports OpenAI, DeepSeek, OpenRouter, and many other providers. Switch between them via configuration without creating separate routes and upstreams for each
3. Built-in AI gateway capabilities: Supports token-level rate limiting, model routing, request/response format transformation, and other AI-gateway-specific features
4. Simplified configuration: Just specify the provider, model name, and API key — the plugin handles the rest

Example equivalent configuration using the ai-proxy plugin:

curl -i http://127.0.0.1:9180/apisix/admin/routes/1 \
  -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" \
  -X PUT -d '{
    "uri": "/anything",
    "plugins": {
      "ai-proxy": {
        "provider": "openrouter",
        "auth": {
          "header": {
            "Authorization": "Bearer YOUR_API_KEY"
          }
        }
      }
    }
  }'

[liuxiaobo007]

Hi @liuxiaobo007, the issue is caused by the default behavior of pass_host in the APISIX upstream configuration.

When you send a request via curl http://127.0.0.1:9080/anything, the client's Host header is 127.0.0.1:9080. The default value of pass_host is "pass", which means APISIX forwards the client's original Host header as-is. This leads to:

1. APISIX sends Host: 127.0.0.1:9080 to the upstream openrouter.ai
2. The TLS SNI (Server Name Indication) during the SSL handshake is also set to 127.0.0.1:9080 (instead of openrouter.ai)
3. OpenRouter's TLS server cannot match a certificate for SNI 127.0.0.1:9080, causing the SSL handshake to fail

Add "pass_host": "node" to the upstream configuration, so APISIX uses the upstream node's domain name (openrouter.ai) as both the Host header and TLS SNI:

curl -i http://127.0.0.1:9180/apisix/admin/routes/1
-H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1"
-X PUT -d '{
"uri": "/anything",
"upstream": {
"type": "roundrobin",
"pass_host": "node",
"nodes": {
"openrouter.ai:443": 1
},
"scheme": "https"
}
}'

Why the ai-proxy Plugin Is Recommended

While the above solution works, if your use case is proxying LLM/AI services, we strongly recommend using the ai-proxy plugin for the following reasons:

1. Automatic connection handling: ai-proxy automatically sets the correct Host, SNI, path, etc. based on the configured provider — no need to manually configure pass_host, scheme, and other low-level settings, avoiding issues like this one
2. Unified multi-model/multi-provider management: Supports OpenAI, DeepSeek, OpenRouter, and many other providers. Switch between them via configuration without creating separate routes and upstreams for each
3. Built-in AI gateway capabilities: Supports token-level rate limiting, model routing, request/response format transformation, and other AI-gateway-specific features
4. Simplified configuration: Just specify the provider, model name, and API key — the plugin handles the rest

Example equivalent configuration using the ai-proxy plugin:

curl -i http://127.0.0.1:9180/apisix/admin/routes/1
-H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1"
-X PUT -d '{
"uri": "/anything",
"plugins": {
"ai-proxy": {
"provider": "openrouter",
"auth": {
"header": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
}'

Thank you very much for your answer. I have another requirement to ask:

Limit-conn plug-in can set the number of concurrent connections for the route. Can this plug-in be fine enough to set the limit of concurrent connections for multiple consumers respectively?

It is ok to enable consumer authentication for this route and limit the number of concurrent connections for this route. How can you set the number of concurrent connections for each consumer of this route separately? If the consumers are not set, there is no limit by default.

For example, the concurrency limit of consumer Alice is 3; Consumer Jack's concurrency limit is 5; Consumer tom's concurrency limit is 2

[Baoyuantop]

Hi @liuxiaobo007, you can bind the plugin to consumers and set different policies for different consumers. See https://docs.api7.ai/hub/limit-conn/#apply-rate-limiting-by-remote-address-and-consumer-name for details.