Does OVMS support Qwen3.5-35B-A3B ?

[shawn9977]

(base) intel@intel-AXMB-D150-3:~/project/qwen35/Qwen3.5-35B-A3B/INT4$ docker run --user $(id -u):$(id -g)
--device /dev/dri/renderD128:/dev/dri/renderD128
--group-add=$(stat -c "%g" /dev/dri/renderD128)
--rm -p 8001:8001
-v $(pwd):$(pwd):rw
openvino/model_server:latest-gpu
--model_path /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
--model_name qwen35-35b
--target_device GPU
--rest_port 8001
[2026-04-24 06:24:18.265][1][serving][info][server.cpp:88] OpenVINO Model Server 2026.1.0.72cc06244
[2026-04-24 06:24:18.265][1][serving][info][server.cpp:89] OpenVINO backend 2026.1.0-21367-63e31528c62-releases/2026/1
[2026-04-24 06:24:18.265][1][serving][info][server.cpp:92] OpenVINO GenAI backend 2026.1.0.0-2957-1dabb8c2255
[2026-04-24 06:24:18.265][1][serving][info][pythoninterpretermodule.cpp:37] PythonInterpreterModule starting
[2026-04-24 06:24:18.265][1][serving][info][pythoninterpretermodule.cpp:40] Initializing python interpreter
[2026-04-24 06:24:18.321][1][serving][info][pythoninterpretermodule.cpp:59] PythonInterpreterModule started
[2026-04-24 06:24:18.359][1][modelmanager][info][modelmanager.cpp:156] Available devices for Open VINO: CPU, GPU
[2026-04-24 06:24:18.359][1][serving][info][capimodule.cpp:40] C-APIModule starting
[2026-04-24 06:24:18.359][1][serving][info][capimodule.cpp:42] C-APIModule started
[2026-04-24 06:24:18.359][1][serving][info][grpcservermodule.cpp:110] GRPCServerModule starting
[2026-04-24 06:24:18.359][1][serving][info][grpcservermodule.cpp:114] GRPCServerModule started
[2026-04-24 06:24:18.359][1][serving][info][grpcservermodule.cpp:115] Port was not set. GRPC server will not be started.
[2026-04-24 06:24:18.359][1][serving][info][httpservermodule.cpp:35] HTTPServerModule starting
[2026-04-24 06:24:18.359][1][serving][info][httpservermodule.cpp:39] Will start 16 REST workers
[2026-04-24 06:24:18.360][50][serving][info][drogon_http_server.cpp:155] Binding REST server to address: 0.0.0.0:8001
[2026-04-24 06:24:18.410][1][serving][info][drogon_http_server.cpp:182] REST server listening on port 8001 with 16 unary threads and 16 streaming threads
[2026-04-24 06:24:18.410][1][serving][info][http_server.cpp:248] API key not provided via --api_key_file or API_KEY environment variable. Authentication will be disabled.
[2026-04-24 06:24:18.410][1][serving][info][httpservermodule.cpp:52] HTTPServerModule started
[2026-04-24 06:24:18.410][1][serving][info][httpservermodule.cpp:53] Started REST server at 0.0.0.0:8001
[2026-04-24 06:24:18.410][1][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-04-24 06:24:18.410][1][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:18.410][1][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
[2026-04-24 06:24:18.410][82][modelmanager][info][modelmanager.cpp:1224] Started model manager thread
[2026-04-24 06:24:18.410][1][serving][info][servablemanagermodule.cpp:55] ServableManagerModule started
[2026-04-24 06:24:18.410][83][modelmanager][info][modelmanager.cpp:1243] Started cleaner thread
[2026-04-24 06:24:19.411][82][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:19.411][82][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
[2026-04-24 06:24:20.411][82][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:20.411][82][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
[2026-04-24 06:24:21.411][82][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:21.411][82][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
[2026-04-24 06:24:22.412][82][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:22.412][82][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
[2026-04-24 06:24:23.412][82][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:23.412][82][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
[2026-04-24 06:24:24.412][82][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:24.412][82][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
[2026-04-24 06:24:25.412][82][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:25.412][82][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
[2026-04-24 06:24:26.413][82][modelmanager][warning][modelmanager.cpp:1488] Expected version directory name to be in number format. Got: model_cache
[2026-04-24 06:24:26.413][82][modelmanager][warning][modelmanager.cpp:1495] No version found for model in path: /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4
^C[2026-04-24 06:24:26.817][1][serving][info][grpcservermodule.cpp:201] GRPCServerModule shutting down
[2026-04-24 06:24:26.817][1][serving][info][grpcservermodule.cpp:211] GRPCServerModule shutdown

[shawn9977]

(base) intel@intel-AXMB-D150-3:~/project/qwen35/Qwen3.5-35B-A3B/INT4$ docker run --user $(id -u):$(id -g) --device /dev/dri/renderD128:/dev/dri/renderD128 --group-add=$(stat -c "%g" /dev/dri/renderD128) --rm -p 9001:9001 -p 8001:8001 -v /home/intel/project/qwen35/Qwen3.5-35B-A3B/INT4:/opt/ml:rw openvino/model_server:latest-gpu --model_path /opt/ml --model_name qwen35-35b --port 9001 --rest_port 8001 --target_device GPU
[2026-04-24 06:49:18.856][1][serving][info][server.cpp:88] OpenVINO Model Server 2026.1.0.72cc06244
[2026-04-24 06:49:18.856][1][serving][info][server.cpp:89] OpenVINO backend 2026.1.0-21367-63e31528c62-releases/2026/1
[2026-04-24 06:49:18.856][1][serving][info][server.cpp:92] OpenVINO GenAI backend 2026.1.0.0-2957-1dabb8c2255
[2026-04-24 06:49:18.856][1][serving][info][pythoninterpretermodule.cpp:37] PythonInterpreterModule starting
[2026-04-24 06:49:18.856][1][serving][info][pythoninterpretermodule.cpp:40] Initializing python interpreter
[2026-04-24 06:49:18.912][1][serving][info][pythoninterpretermodule.cpp:59] PythonInterpreterModule started
[2026-04-24 06:49:18.989][1][modelmanager][info][modelmanager.cpp:156] Available devices for Open VINO: CPU, GPU
[2026-04-24 06:49:18.990][1][serving][info][capimodule.cpp:40] C-APIModule starting
[2026-04-24 06:49:18.990][1][serving][info][capimodule.cpp:42] C-APIModule started
[2026-04-24 06:49:18.990][1][serving][info][grpcservermodule.cpp:110] GRPCServerModule starting
[2026-04-24 06:49:18.990][1][serving][info][grpcservermodule.cpp:137] Binding gRPC server to address: 0.0.0.0:9001
[2026-04-24 06:49:18.990][1][serving][info][grpcservermodule.cpp:192] GRPCServerModule started
[2026-04-24 06:49:18.990][1][serving][info][grpcservermodule.cpp:193] Started gRPC server on port 9001
[2026-04-24 06:49:18.990][1][serving][info][httpservermodule.cpp:35] HTTPServerModule starting
[2026-04-24 06:49:18.990][1][serving][info][httpservermodule.cpp:39] Will start 16 REST workers
[2026-04-24 06:49:18.991][54][serving][info][drogon_http_server.cpp:155] Binding REST server to address: 0.0.0.0:8001
[2026-04-24 06:49:19.042][1][serving][info][drogon_http_server.cpp:182] REST server listening on port 8001 with 16 unary threads and 16 streaming threads
[2026-04-24 06:49:19.042][1][serving][info][http_server.cpp:248] API key not provided via --api_key_file or API_KEY environment variable. Authentication will be disabled.
[2026-04-24 06:49:19.042][1][serving][info][httpservermodule.cpp:52] HTTPServerModule started
[2026-04-24 06:49:19.042][1][serving][info][httpservermodule.cpp:53] Started REST server at 0.0.0.0:8001
[2026-04-24 06:49:19.042][1][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-04-24 06:49:19.042][1][serving][info][model.cpp:42] Getting model from /opt/ml
[2026-04-24 06:49:19.042][1][serving][info][model.cpp:49] Model downloaded to /opt/ml
[2026-04-24 06:49:19.042][1][serving][info][model.cpp:149] Will add model: qwen35-35b; version: 1 ...
[2026-04-24 06:49:19.042][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model qwen35-35b status change. New status: ( "state": "START", "error_code": "OK" )
[2026-04-24 06:49:19.042][1][serving][info][modelinstance.cpp:1136] Loading model: qwen35-35b, version: 1, from path: /opt/ml/1, with target device: GPU ...
[2026-04-24 06:49:19.042][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model qwen35-35b status change. New status: ( "state": "START", "error_code": "OK" )
[2026-04-24 06:49:19.042][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model qwen35-35b status change. New status: ( "state": "LOADING", "error_code": "OK" )
[2026-04-24 06:49:19.066][1][modelmanager][info][modelinstance.cpp:671] Input name: Parameter_435122; mapping_name: Parameter_435122; shape: (-1); precision: STRING; layout: N...
[2026-04-24 06:49:19.066][1][modelmanager][info][modelinstance.cpp:734] Output name: input_ids; mapping_name: input_ids; shape: (-1,-1); precision: I64; layout: N...
[2026-04-24 06:49:19.066][1][modelmanager][info][modelinstance.cpp:734] Output name: attention_mask; mapping_name: attention_mask; shape: (-1,-1); precision: I64; layout: N...
[2026-04-24 06:49:19.105][1][modelmanager][error][modelinstance.cpp:920] Cannot compile model into target device; error: Exception from src/inference/src/cpp/core.cpp:118:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/intel_gpu/src/plugin/program_builder.cpp:223:
Operation: StringTensorUnpack_435123 of type StringTensorUnpack(opset15) is not supported

; model: qwen35-35b; version: 1; device: GPU
[2026-04-24 06:49:19.105][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model qwen35-35b status change. New status: ( "state": "LOADING", "error_code": "UNKNOWN" )
[2026-04-24 06:49:19.105][1][serving][error][model.cpp:154] Error occurred while loading model: qwen35-35b; version: 1; error: Cannot compile model into target device
[2026-04-24 06:49:19.105][1][modelmanager][error][modelmanager.cpp:1506] Error occurred while loading model: qwen35-35b versions; error: Cannot compile model into target device
[2026-04-24 06:49:19.105][1][serving][info][model.cpp:197] Will clean up model: qwen35-35b; version: 1 ...
[2026-04-24 06:49:19.105][1][serving][info][model.cpp:89] Updating default version for model: qwen35-35b, from: 0
[2026-04-24 06:49:19.105][1][serving][info][model.cpp:101] Model: qwen35-35b will not have default version since no version is available.
[2026-04-24 06:49:19.105][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model qwen35-35b status change. New status: ( "state": "LOADING", "error_code": "UNKNOWN" )
[2026-04-24 06:49:19.105][1][modelmanager][error][modelmanager.cpp:184] Couldn't start model manager
[2026-04-24 06:49:19.105][1][serving][error][servablemanagermodule.cpp:58] ovms::ModelManager::Start() Error: Cannot compile model into target device
[2026-04-24 06:49:19.105][1][serving][info][grpcservermodule.cpp:201] GRPCServerModule shutting down
[2026-04-24 06:49:19.105][1][serving][info][grpcservermodule.cpp:206] Shutdown gRPC server
[2026-04-24 06:49:19.105][1][serving][info][grpcservermodule.cpp:211] GRPCServerModule shutdown
[2026-04-24 06:49:19.105][1][serving][info][httpservermodule.cpp:59] HTTPServerModule shutting down
[2026-04-24 06:49:19.107][1][serving][info][httpservermodule.cpp:64] Shutdown HTTP server
[2026-04-24 06:49:19.107][1][serving][info][servablemanagermodule.cpp:65] ServableManagerModule shutting down
[2026-04-24 06:49:19.107][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model qwen35-35b status change. New status: ( "state": "UNLOADING", "error_code": "OK" )
[2026-04-24 06:49:19.107][1][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model qwen35-35b status change. New status: ( "state": "END", "error_code": "OK" )
[2026-04-24 06:49:19.108][1][serving][info][servablemanagermodule.cpp:71] ServableManagerModule shutdown
[2026-04-24 06:49:19.108][1][serving][info][pythoninterpretermodule.cpp:70] PythonInterpreterModule shutting down
[2026-04-24 06:49:19.108][1][serving][info][pythoninterpretermodule.cpp:74] PythonInterpreterModule shutdown
Python version:
3.12.3 (main, Mar 3 2026, 12:15:18) [GCC 13.3.0]
Python sys.path output:
['', '/ovms/lib/python', '/ovms/python_deps', '/lib/python312.zip', '/lib/python3.12', '/lib/python3.12/lib-dynload']
[2026-04-24 06:49:19.110][1][serving][info][capimodule.cpp:50] C-APIModule shutting down
[2026-04-24 06:49:19.110][1][serving][info][capimodule.cpp:52] C-APIModule shutdown

[dkalinowski]

Qwen3.5 model enablement is in progress, you can track it here: openvinotoolkit/openvino.genai#3717