Title: stream=True only returns output from basemodel, LoRA adapter is ignored.

[HARISHSENTHIL]

System Info

When using Lorax with a LoRA adapter via the /v1/chat/completions endpoint, the adapter works as expected when "stream": false.

However, when I set "stream": true, the response is clearly from the base model only, and the adapter (adapter_name) appears to be ignored.

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Works:
{
"model": "Mistral-7B-Instruct-v0.1",
"adapter_name": "Medical-Insights-QA",
"stream": false,
"messages": [
{"role": "user", "content": "What are symptoms of cancer?"}
]
}
Broken (stream: true only uses base model):
{
"model": "Mistral-7B-Instruct-v0.1",
"adapter_name": "Medical-Insights-QA",
"stream": true,
"messages": [
{"role": "user", "content": "What are symptoms of cancer?"}
]
}

Expected behavior

When using the /v1/chat/completions endpoint with "stream": true, I expect the model to generate streamed responses using the specified LoRA adapter (adapter_name) — just like it does when "stream": false.
The adapter should influence generation in both streaming and non-streaming modes, resulting in consistent behavior and outputs aligned with the fine-tuned model.

[rehan243]

fwiw, we ran into something similar when streaming with a LoRA adapter on Mistral (we were using a custom adapter, not Medical-Insights-QA). The issue for us was that the streaming codepath wasn’t applying the LoRA weights correctly — turns out the base model gets loaded separately for non-streaming vs streaming calls, and the adapter wasn’t being merged in for the streaming handler.

Quick fix for us was ensuring the peft_model.merge_and_unload() step ran right after loading the adapter AND before the response generator kicked off:

from peft import PeftModel

model = load_model("Mistral-7B-Instruct-v0.1")
adapter = PeftModel.from_pretrained(model, "Medical-Insights-QA")
adapter.merge_and_unload()  # forces the LoRA weights into the base model

After that, stream: true started using the adapter as expected. Not sure if you're doing something similar, but might be worth checking how/where the adapter merge happens in the streaming flow.