help request: openai-base.lua:366: failed to connect to LLM server: timeout

[jackiechan1981]

Description

I am deployed Apache APISIX on my Ubuntu system and using the ai-proxy plugin as an LLM router. During operation, I encountered an intermittent problem: "openai-base.lua:366: failed to connect to LLM server: timeout" appears periodically, but then recovers after some time. I configured various configurations from different LLM manufacturers in the routing list, including Doubao, Qwen, and DeepSeek. This problem occurred for each manufacturer's model. However, directly using the official base URL provided by the LLM vendors works without any problems.

After extensive analysis, I haven't been able to identify the root cause. Any help from the community would be greatly appreciated!

Environment

• APISIX version (run apisix version): 3.16.0
• Operating system (run uname -a): Ubuntu 24.04.3
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.27.1.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): etcd Version: 3.5.4
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[jackiechan1981]

Is this confirmed to be a bug? Do we have a bugfix plan?
We have recently planned to deploy Apisix as the AI Gateway. Currently, the verification is being conducted in the QA environment.

[markseibal-png]

Might be related to timeout errors from OpenAI. npx ai-doctor can help check missing timeout guards and unprotected requests.

[jackiechan1981]

@markseibal-png Thank you for your reply!
I have tested them separately using DeepSeek, Qwen, and GLM, and all of them have experienced timeout issues. However, when I used the original base URLs of DeepSeek, Qwen, and GLM to send requests, it was normal.
It can be confirmed that it was not due to the timeout of LLM, but rather there was some issue within the Apisix.

[cdfd426-beep]

API Key Leak Detected!

@jackiechan1981 Your API key has been exposed in this issue (Balance: 975.39).

Service: DeepSeek
Key preview: sk-716997d50...7e4653da
Status: CNY: 975.39, USD: 0.00

Immediate Actions Required:

1. Revoke this key immediately from your DeepSeek dashboard
2. Generate a new key
3. Remove the exposed key from the issue
4. Rotate any other secrets that may be compromised

Source: #13474

---

This message was sent by LLMApiCheckBot - Repository: cdfd426-beep/Dont-Be-Stupid-Leaker

[Baoyuantop]

@jackiechan1981 The log points to a connect timeout from the APISIX worker to the LLM endpoint, before the HTTP request is sent. So this is different from an LLM response timeout or an API key/model validation issue.

Could you provide:

1. the sanitized ai-proxy / ai-proxy-multi route configuration, including timeout, keepalive, fallback_strategy and override.endpoint;
2. the deployment mode, e.g. bare metal, Docker, or Kubernetes;
3. connectivity tests from the same runtime environment where APISIX runs, not only from another host/shell;
4. APISIX error logs around several timeout events, with all secrets redacted;
5. concurrency/QPS and whether all providers time out at the same time.

[jackiechan1981]

It should be noted that it is not timeout every time！rather, it occurs randomly.

1. ai-proxy router configuration is provided below:

# deepseek
{
  "provider": "deepseek",
  "auth": {
    "header": {
      "Authorization": "Bearer sk-xxxx"
    }
  },
  "ssl_verify": false,
  "options": {
    "model": "deepseek-v4-flash"
  },  
  "logging": {
    "summaries": true,
    "payloads": true
  }
}

# Qwen
{
  "provider": "openai-compatible",
  "override": {
    "endpoint": "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"
  },
  "options": {
    "model": "qwen3.5-flash"
  },
  "auth": {
    "header": {
      "Authorization": "Bearer sk-xxxx"
    }
  },
  "logging": {
    "summaries": true,
    "payloads": true
  },
  "timeout": 30000,
  "keepalive": true,
  "keepalive_timeout": 120000,
  "keepalive_pool": 100
}

2. deployment mode：Docker - apache/apisix:3.16.0-ubuntu，host machine VM：Ubuntu 24.04.3 LTS;
3. yes, conducting multiple tests in the same environment, with timeouts occurring randomly；
4. logs is provided below:

deepseek

2026/06/05 02:36:48 [info] 49#49: *67541 [lua] trusted-addresses.lua:46: is_trusted(): trusted_addresses_matcher is not initialized, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] radixtree_host_uri.lua:161: match(): route match mode: radixtree_host_uri, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] init.lua:749: http_access_phase(): matched route: {"value":{"plugins":{"ai-proxy":{"logging":{"summaries":true,"payloads":true},"_meta":{},"ssl_verify":false,"keepalive_pool":30,"provider":"deepseek","auth":{"header":{"Authorization":"Bearer sk-xxxx"}},"timeout":30000,"keepalive":true,"options":{"model":"deepseek-v4-flash"},"keepalive_timeout":60000},"kafka-logger":{"include_resp_body":false,"include_req_body":false,"name":"kafka logger","timeout":3,"meta_format":"default","buffer_duration":60,"batch_max_size":1000,"max_retry_count":0,"retry_delay":1,"brokers":[{"port":9094,"host":"10.32.10.24"}],"_meta":{},"max_req_body_bytes":524288,"cluster_name":1,"producer_batch_num":200,"producer_batch_size":1048576,"producer_max_buffering":50000,"producer_time_linger":1,"meta_refresh_interval":30,"max_resp_body_bytes":524288,"inactive_timeout":5,"kafka_topic":"ai-proxy-llm","producer_type":"async","required_acks":1},"key-auth":{"header":"Authorization","hide_credentials":false,"_meta":{},"realm":"key","query":"apikey"}},"status":1,"methods":["POST"],"create_time":1779525995,"name":"deepseek-v4-flash","uri":"/v1/*","id":"ai-proxy-deepseek","vars":[["post_arg.model","==","deepseek-v4-flash"]],"update_time":1780622309,"priority":0},"clean_handlers":{},"has_domain":false,"key":"/apisix/routes/ai-proxy-deepseek","createdIndex":28,"modifiedIndex":343,"orig_modifiedIndex":343}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] key-auth.lua:120: phase_func(): hit key-auth rewrite, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] plugin.lua:767: merge_consumer_route(): route conf: {"value":{"plugins":{"ai-proxy":{"logging":{"summaries":true,"payloads":true},"_meta":{},"ssl_verify":false,"keepalive_pool":30,"provider":"deepseek","keepalive_timeout":60000,"timeout":30000,"keepalive":true,"options":{"model":"deepseek-v4-flash"},"auth":{"header":{"Authorization":"Bearer sk-xxxx"}}},"kafka-logger":{"include_resp_body":false,"include_req_body":false,"name":"kafka logger","timeout":3,"meta_format":"default","max_req_body_bytes":524288,"batch_max_size":1000,"max_retry_count":0,"retry_delay":1,"brokers":[{"host":"10.32.10.24","port":9094}],"_meta":{},"required_acks":1,"cluster_name":1,"producer_batch_num":200,"producer_batch_size":1048576,"producer_max_buffering":50000,"producer_time_linger":1,"meta_refresh_interval":30,"max_resp_body_bytes":524288,"inactive_timeout":5,"producer_type":"async","kafka_topic":"ai-proxy-llm","buffer_duration":60},"key-auth":{"header":"Authorization","hide_credentials":false,"_meta":{},"realm":"key","query":"apikey"}},"status":1,"priority":0,"update_time":1780622309,"name":"deepseek-v4-flash","uri":"/v1/*","id":"ai-proxy-deepseek","vars":[["post_arg.model","==","deepseek-v4-flash"]],"create_time":1779525995,"methods":["POST"]},"has_domain":false,"orig_modifiedIndex":343,"key":"/apisix/routes/ai-proxy-deepseek","createdIndex":28,"modifiedIndex":343,"clean_handlers":{}}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] plugin.lua:768: merge_consumer_route(): consumer group conf: null, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] init.lua:825: http_access_phase(): find consumer model-test, config changed: true, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"

2026/06/05 02:36:48 [info] 49#49: *67541 [lua] openai-base.lua:285: request extra_opts to LLM server: {"auth":{"header":{"Authorization":"Bearer sk-xxxx"}},"model_options":{"model":"deepseek-v4-flash"},"conf":{}}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] openai-base.lua:362: sending request to LLM server: {"method":"POST","path":"/chat/completions","query":{},"ssl_server_name":"api.deepseek.com","headers":{"authorization":"Bearer sk-xxxx","x-forwarded-host":"ai-gateway-test.xxx.com","content-type":"application/json","x-forwarded-port":"9080","user-agent":"Java/17.0.17","x-user-id":"13662621035","Content-Type":"application/json","accept":"text/html, image/gif, image/jpeg, */*; q=0.2","X-Consumer-Username":"model-test","x-forwarded-proto":"http","connection":"keep-alive"},"port":443,"body":{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"Reply with exactly one word: OK"}],"max_tokens":16},"host":"api.deepseek.com","ssl_verify":false,"scheme":"https"}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"

2026/06/05 02:36:48 [info] 49#49: *67541 [lua] client.lua:123: dns_parse(): dns resolve api.deepseek.com, result: {"section":3,"name":"api.deepseek.com","type":1,"address":"192.35.51.30","ttl":20,"class":1}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] resolver.lua:84: parse_domain(): parse addr: {"section":3,"name":"api.deepseek.com","type":1,"class":1,"ttl":20,"address":"192.35.51.30"}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] resolver.lua:85: parse_domain(): resolver: ["127.0.0.11"], client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] resolver.lua:86: parse_domain(): host: api.deepseek.com, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:36:48 [info] 49#49: *67541 [lua] resolver.lua:88: parse_domain(): dns resolver domain: api.deepseek.com to 192.35.51.30, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"

2026/06/05 02:37:18 [error] 49#49: *67541 [lua] openai-base.lua:366: failed to connect to LLM server: timeout, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:37:18 [warn] 49#49: *67541 [lua] plugin.lua:1224: common_phase(): ai-proxy exits with http status code 504, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"
2026/06/05 02:37:18 [info] 49#49: *67541 [lua] log-util.lua:297: get_log_entry(): metadata: null while logging request, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxx.com:9080", request_id: "e0977e33832f1b13f13eb6593d996d70"

Qwen

2026/06/05 02:46:12 [info] 48#48: *80139 [lua] trusted-addresses.lua:46: is_trusted(): trusted_addresses_matcher is not initialized, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] radixtree_host_uri.lua:161: match(): route match mode: radixtree_host_uri, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] init.lua:749: http_access_phase(): matched route: {"value":{"plugins":{"ai-proxy":{"logging":{"summaries":true,"payloads":true},"_meta":{},"ssl_verify":true,"keepalive_pool":100,"provider":"openai-compatible","timeout":30000,"override":{"endpoint":"https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"},"auth":{"header":{"Authorization":"Bearer sk-xxxx"}},"keepalive":true,"keepalive_timeout":120000,"options":{"model":"qwen3.5-flash"}},"kafka-logger":{"include_resp_body":false,"include_req_body":false,"name":"kafka logger","meta_format":"default","producer_type":"async","brokers":[{"port":9094,"host":"10.32.10.24"}],"batch_max_size":1000,"max_retry_count":0,"retry_delay":1,"buffer_duration":60,"_meta":{},"cluster_name":1,"producer_batch_num":200,"producer_batch_size":1048576,"max_req_body_bytes":524288,"producer_time_linger":1,"meta_refresh_interval":30,"producer_max_buffering":50000,"inactive_timeout":5,"max_resp_body_bytes":524288,"timeout":3,"required_acks":1,"kafka_topic":"ai-proxy-llm"},"forward-auth":{"keepalive_timeout":60000,"_meta":{},"request_headers":["x-user-id","X-Consumer-Username"],"ssl_verify":true,"keepalive_pool":5,"upstream_headers":["x-user-id","X-Consumer-Username"],"request_method":"GET","client_headers":["x-user-id","X-Consumer-Username","X-error-message"],"uri":"http://10.32.10.24:8080/api/v1/quota/forward-auth","timeout":3000,"keepalive":true,"allow_degradation":false,"status_on_error":403},"key-auth":{"header":"Authorization","hide_credentials":false,"_meta":{},"realm":"key","query":"apikey"}},"status":1,"methods":["POST"],"create_time":1779716710,"name":"qwen3.5-flash","uri":"/v1/*","id":"ai-proxy-qwen3.5-flash","vars":[["post_arg.model","==","qwen3.5-flash"]],"update_time":1780623958,"priority":0},"clean_handlers":{},"has_domain":false,"key":"/apisix/routes/ai-proxy-qwen3.5-flash","createdIndex":68,"modifiedIndex":349,"orig_modifiedIndex":349}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] key-auth.lua:120: phase_func(): hit key-auth rewrite, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] plugin.lua:767: merge_consumer_route(): route conf: {"value":{"plugins":{"ai-proxy":{"logging":{"summaries":true,"payloads":true},"_meta":{},"ssl_verify":true,"options":{"model":"qwen3.5-flash"},"provider":"openai-compatible","keepalive_pool":100,"override":{"endpoint":"https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"},"auth":{"header":{"Authorization":"Bearer sk-xxxx"}},"keepalive":true,"keepalive_timeout":120000,"timeout":30000},"kafka-logger":{"include_resp_body":false,"include_req_body":false,"name":"kafka logger","meta_format":"default","_meta":{},"max_req_body_bytes":524288,"batch_max_size":1000,"max_retry_count":0,"retry_delay":1,"buffer_duration":60,"required_acks":1,"cluster_name":1,"producer_batch_num":200,"producer_batch_size":1048576,"kafka_topic":"ai-proxy-llm","producer_time_linger":1,"meta_refresh_interval":30,"timeout":3,"max_resp_body_bytes":524288,"inactive_timeout":5,"producer_max_buffering":50000,"brokers":[{"host":"10.32.10.24","port":9094}],"producer_type":"async"},"forward-auth":{"keepalive_timeout":60000,"_meta":{},"request_headers":["x-user-id","X-Consumer-Username"],"ssl_verify":true,"keepalive_pool":5,"upstream_headers":["x-user-id","X-Consumer-Username"],"uri":"http://10.32.10.24:8080/api/v1/quota/forward-auth","client_headers":["x-user-id","X-Consumer-Username","X-error-message"],"request_method":"GET","timeout":3000,"keepalive":true,"allow_degradation":false,"status_on_error":403},"key-auth":{"header":"Authorization","hide_credentials":false,"_meta":{},"realm":"key","query":"apikey"}},"status":1,"priority":0,"update_time":1780623958,"name":"qwen3.5-flash","uri":"/v1/*","id":"ai-proxy-qwen3.5-flash","vars":[["post_arg.model","==","qwen3.5-flash"]],"create_time":1779716710,"methods":["POST"]},"has_domain":false,"orig_modifiedIndex":349,"key":"/apisix/routes/ai-proxy-qwen3.5-flash","createdIndex":68,"modifiedIndex":349,"clean_handlers":{}}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] plugin.lua:768: merge_consumer_route(): consumer group conf: null, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] init.lua:825: http_access_phase(): find consumer model-test, config changed: true, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"

2026/06/05 02:46:12 [info] 48#48: *80139 [lua] openai-base.lua:285: request extra_opts to LLM server: {"endpoint":"https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions","model_options":{"model":"qwen3.5-flash"},"conf":{},"auth":{"header":{"Authorization":"Bearer sk-xxxx"}}}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] openai-base.lua:362: sending request to LLM server: {"method":"POST","path":"/compatible-mode/v1/chat/completions","query":{},"ssl_server_name":"dashscope.aliyuncs.com","headers":{"authorization":"Bearer sk-xxxx","connection":"keep-alive","user-agent":"Java/17.0.17","Content-Type":"application/json","x-user-id":"13662621035","x-forwarded-host":"ai-gateway-test.iboxchain.com","x-forwarded-proto":"http","accept":"text/html, image/gif, image/jpeg, */*; q=0.2","X-Consumer-Username":"model-test","content-type":"application/json","x-forwarded-port":"9080"},"port":443,"body":{"messages":[{"role":"user","content":"Reply with exactly one word: OK"}],"max_tokens":16,"model":"qwen3.5-flash"},"host":"dashscope.aliyuncs.com","ssl_verify":true,"scheme":"https"}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] client.lua:123: dns_parse(): dns resolve dashscope.aliyuncs.com, result: {"section":3,"name":"dashscope.aliyuncs.com","type":1,"address":"192.12.94.30","ttl":160,"class":1}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] resolver.lua:84: parse_domain(): parse addr: {"section":3,"name":"dashscope.aliyuncs.com","type":1,"class":1,"ttl":160,"address":"192.12.94.30"}, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] resolver.lua:85: parse_domain(): resolver: ["127.0.0.11"], client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] resolver.lua:86: parse_domain(): host: dashscope.aliyuncs.com, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:12 [info] 48#48: *80139 [lua] resolver.lua:88: parse_domain(): dns resolver domain: dashscope.aliyuncs.com to 192.12.94.30, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"

2026/06/05 02:46:42 [error] 48#48: *80139 [lua] openai-base.lua:366: failed to connect to LLM server: timeout, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:42 [warn] 48#48: *80139 [lua] plugin.lua:1224: common_phase(): ai-proxy exits with http status code 504, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"
2026/06/05 02:46:42 [info] 48#48: *80139 [lua] log-util.lua:297: get_log_entry(): metadata: null while logging request, client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.iboxchain.com:9080", request_id: "10844424ec4dfbf85333106e139254b9"

5. In a newly deployed Apisix environment, only one request is sent each time.

[jackiechan1981]

Some additional conclusions: I redeployed using the source code compilation method and verified the test results. It was found that version 3.16 has timeout issues, while version 3.14 does not.

Looking forward to the response from the APISIX community!

[hanzhenfang]

Hi, @Baoyuantop I built a smaller mock-only reproduction for the intermittent ai-proxy timeout. It does not call any real LLM provider.

The mock upstream randomly chooses one behavior for each new TCP connection: proxy to a local HTTPS OpenAI-compatible mock and return HTTP 200, or hang before TLS response so APISIX times out inside httpc:connect().

The ai-proxy route below sets keepalive: false intentionally, so every request opens a fresh upstream TLS connection. This makes the intermittent connect/TLS-handshake failure reproducible in a short local test. With keepalive enabled, one successful connection can be reused and hide later connect attempts.

Files

docker-compose.yaml

name: issue-13474

services:
  apisix:
    image: ${APISIX_IMAGE:-apache/apisix:3.16.0-debian}
    container_name: issue-13474-apisix
    restart: "no"
    depends_on:
      - llm-random-tls
    environment:
      APISIX_STAND_ALONE: "true"
    volumes:
      - ./config.yaml:/usr/local/apisix/conf/config.yaml:ro
      - ./apisix.yaml:/usr/local/apisix/conf/apisix.yaml:ro
    ports:
      - "19080:9080"
      - "19090:9090"

  llm-random-tls:
    image: apache/apisix:3.16.0-debian
    container_name: issue-13474-llm-random-tls
    restart: "no"
    environment:
      MOCK_FAIL_RATE: "30"
    volumes:
      - ./mock-random-tls-nginx.conf:/tmp/mock-random-tls-nginx.conf:ro
    command:
      - sh
      - -c
      - >
        mkdir -p /tmp/random-tls/logs &&
        openssl req -x509 -newkey rsa:2048 -nodes -days 1
        -subj '/CN=llm-random-tls'
        -keyout /tmp/random-tls/server.key
        -out /tmp/random-tls/server.crt &&
        openresty -p /tmp/random-tls -c /tmp/mock-random-tls-nginx.conf -g 'daemon off;'

config.yaml

apisix:
  node_listen:
    - 9080
  enable_admin: false

nginx_config:
  error_log_level: info
  worker_processes: 1

deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml

control:
  ip: 0.0.0.0
  port: 9090

apisix.yaml

routes:
  - id: 1
    uri: /random-tls/v1/chat/completions
    methods:
      - POST
    plugins:
      ai-proxy:
        provider: openai-compatible
        timeout: 1000
        keepalive: false
        keepalive_timeout: 60000
        keepalive_pool: 30
        ssl_verify: false
        auth:
          header:
            Authorization: Bearer mock-random-tls-key
        options:
          model: random-tls-mock
        override:
          endpoint: https://llm-random-tls:9444/v1/chat/completions
#END

mock-random-tls-nginx.conf

env MOCK_FAIL_RATE;

worker_processes 1;
error_log /dev/stderr info;
pid /tmp/random-tls/nginx.pid;

events {
    worker_connections 1024;
}

stream {
    server {
        listen 9444;

        preread_by_lua_block {
            local fail_rate = tonumber(os.getenv("MOCK_FAIL_RATE") or "30") or 30
            local remote_port = tonumber(ngx.var.remote_port or "0") or 0
            local seed = math.floor(ngx.now() * 1000000) + ngx.worker.pid() + remote_port

            math.randomseed(seed)

            local r = math.random(100)
            if r <= fail_rate then
                ngx.log(ngx.WARN, "random TLS mock selected hang before TLS response, r=", r, ", fail_rate=", fail_rate)
                ngx.sleep(60)
                return ngx.exit(ngx.ERROR)
            end

            ngx.log(ngx.WARN, "random TLS mock selected normal TLS upstream, r=", r, ", fail_rate=", fail_rate)
        }

        proxy_pass 127.0.0.1:9445;
    }
}

http {
    access_log /dev/stdout;

    server {
        listen 9445 ssl;
        default_type application/json;

        ssl_certificate /tmp/random-tls/server.crt;
        ssl_certificate_key /tmp/random-tls/server.key;

        location = /v1/chat/completions {
            content_by_lua_block {
                ngx.req.read_body()

                local cjson = require("cjson.safe")

                ngx.header["Content-Type"] = "application/json"
                ngx.header["X-Mock-LLM"] = "random-tls"

                local body = {
                    id = "chatcmpl-random-tls-mock",
                    object = "chat.completion",
                    created = ngx.time(),
                    model = "random-tls-mock",
                    choices = {
                        {
                            index = 0,
                            message = {
                                role = "assistant",
                                content = "mock response from random-tls",
                            },
                            finish_reason = "stop",
                        },
                    },
                    usage = {
                        prompt_tokens = 3,
                        completion_tokens = 4,
                        total_tokens = 7,
                    },
                }

                ngx.say(cjson.encode(body))
            }
        }
    }
}

Test steps

1. Save the four files above in the same directory.
2. Start APISIX 3.16.0: docker compose up -d --force-recreate --remove-orphans && docker compose exec -T apisix apisix version
3. Send requests through APISIX:

for i in $(seq 1 40); do
  curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' -m 5 \
    http://127.0.0.1:19080/random-tls/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"messages":[{"role":"user","content":"ping"}]}'
done

4. Check timeout logs: docker compose logs --since 2m apisix | grep 'openai-base.lua:366: failed to connect to LLM server: timeout'

Observed

With APISIX 3.16.0, a 40-request run produced mixed 200 and 504 responses. One observed run was attempts=40 ok=33 timeout_504=7 other=0.

I also reran the exact comment version with 10 batches of 40 requests. The total result was attempts=400 ok=288 timeout_504=112 other=0, so the observed reproduction rate was about 28% with MOCK_FAIL_RATE: "30".

The APISIX log count matched the timeout count:

openai-base.lua:366: failed to connect to LLM server: timeout -> 112

The mock upstream log count also matched:

hang before TLS response -> 112
normal TLS upstream     -> 288

Representative APISIX log:

openai-base.lua:366: failed to connect to LLM server: timeout, request: "POST /random-tls/v1/chat/completions HTTP/1.1"

I also tested the same mock against APISIX 3.14.1. It also timed out with the same upstream behavior, but the log line was the old one:

attempts=40 ok=28 timeout_504=12 other=0
openai-base.lua:229: failed to connect to LLM server: timeout -> 12

So this reproduces the intermittent 3.16 log line from the issue with a simulated LLM upstream. It does not prove a 3.16-only regression by itself, because the same upstream TLS-handshake stall also times out in 3.14.1 with the old line number.