Feature requests and TODO

[SearchSavior]

Hi everyone,

Like llama.cpp OpenArc is meant to be infrastructure for projects using intel hardware. I want to take temperature from our users- what's missing? A big question that I need help to answer.

Short term:

• meatposes and WizardlyBump shared working docker configs so that will be fleshed out.
• speculative decoding for llm, along with reccomended pairings and updates to
• resolve Tool call ID error causing model unload - responding after tool use broken? #48
• harden unfinished test suite

Longer term New features:

• topk logprobs for llm and vlm
• benchmarking for vlm. Think a range of default images as a test suite.
• nncf quantization diagnostic "toolkit"
• ContinuousBatchingPipeline ie paged attention like vllm (!)
• lora hotswap

[meatposes]

Putting aside my project manager hat for a minute, there are some QOL items I would like to see:

1. Dynamic model loading (and unloading) via a preferred model flag or favorite identifier
   Currently only loaded model(s) is presented in /v1/models. While I don't personally want my entire config presented to something like openwebUI, the ability for those models to be presented and switched automatically via the LLM tooling would be nice. How exactly this works would need to be explored. Some other model servers handle this reasonably well. Would also need to consider how automatic or configurable unloading behavior works here.
2. A way to flag or highlight preferred or favorite configurations. This could be used in parallel with above to present only a few items in /v1/models AND/OR to provide a fast swap option. Maybe this is a me problem with ~30-40 different configurations currently that I am using for testing
3. Unload all models switch/API and alternatively/in concert an internal reset switch/API call that clears everything out and starts it up again clean
4. better configuration validation or testing/probing

If you can help guide your long term vision for the project I have some other ideas, but I have limited these to core functionality

[feilonguu]

When will support for the Qwen-3 VL 8B multimodal large model be available? I deployed the Qwen-3 VL 8B model using the latest version of OpenVINO, but there is no tool to deploy an API service. Thank you.

[SearchSavior]

Optimum team are implementing, and once support lands in openvino genai I will make sure it works in openarc. huggingface/optimum-intel#1551

[DrazorV]

From the POV of someone very inexperienced in LLM, I am struggling with documentation. I have managed to setup the project, load some models and integrate openArc to home assistant but after 2 prompts it crashes the LLM and I can't find anything online. I have not yet found how to disable/enable reasoning, change the chat-template (not even sure if it has one), I still hasn't managed to create a systemctl service to keep it running 24/7 and the list of all the env variables.

The project is great, it gave me very good performance when tested on bench, it is easier to setup than Llama.cpp vulkan/sycl. The only thing that i had to do extra that it wasn't mentioned in the setup guide was to install the Intel drivers for Ubuntu 24.04 because it wouldn't recognize the gpu at first.

[DrazorV]

I managed to setup the service and create a command outside the venv like so:

cat >/etc/systemd/system/openarc.service <<'EOF'
[Unit]
Description=OpenArc Inference Server (venv)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/root/OpenArc

# Server auth key
Environment=OPENARC_API_KEY=PLACEHOLDER_API

# Run using the venv openarc (no "activate" needed)
Environment=VIRTUAL_ENV=/root/OpenArc/.venv
Environment=PATH=/root/OpenArc/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

ExecStart=/root/OpenArc/.venv/bin/openarc serve start

Restart=on-failure
RestartSec=2
KillSignal=SIGINT
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now openarc

ln -sf /root/OpenArc/.venv/bin/openarc /usr/local/bin/openarc

and test it:

which openarc
openarc status
openarc list