Skip to content

fix(server): reset request settings to the model's defaults on every request - #978

Open
NVolcz wants to merge 1 commit into
ROCm:mainfrom
NVolcz:reset-request-settings
Open

NVolcz wants to merge 1 commit into
ROCm:mainfrom
NVolcz:reset-request-settings

Conversation

@NVolcz

@NVolcz NVolcz commented Nov 7, 2026

Copy link
Copy Markdown

The server applies a request's settings (think, reasoning_effort, temperature, top_p, top_k, min_p, the penalties) to the shared engine and never resets them. A setting one request gives stays in place for every later request that leaves it out, from any client. For example, after one request with "think": true, requests without think still render <|think|> and the model keeps reasoning. The docs warned about this ("Settings stick").

The engine now remembers its sampler config and whether thinking is on once the model is loaded, and the server puts them back before applying each request's settings, on the chat endpoints and on /api/generate. A setting a request leaves out falls back to the model's default. Because the sampler is recreated, every request also starts with an empty penalty window.

The docs, a code comment and the API tests' notes now describe the new behaviour, and test_reasoning.think_does_not_carry_over checks it.

Qwen3_5_Omni::set_sampler did the same as AutoModel::set_sampler, so it is removed; otherwise Qwen3.5-Omni would skip remembering its sampler config.

The reasoning_effort levels of GPT-OSS and Hy-Memory, conversational_mode and image-max-tokens are stored in model-specific fields and still stay in effect until the model is reloaded.

I noticed this while reading the prompts flm renders, with a small debug branch that logs every prompt and raw output to a file (log-prompt, FLM_LOG_PROMPT=<file>): a request without think still had <|think|> in its prompt.

…request

A setting one request gave, such as think or temperature, stayed on the shared engine and
applied to every later request that left it out, from any client: after one request with
think: true, requests without it still got thinking. The docs warned about this ("Settings
stick").

The engine remembers its sampler config and whether thinking is on once the model is loaded,
and the server puts them back before applying a request's settings, on the chat endpoints and
on /api/generate. As the sampler is recreated, every request also starts with an empty penalty
window. Qwen3.5-Omni's set_sampler override did the same as the base one, so it is gone.

The reasoning_effort levels of GPT-OSS and Hy-Memory, conversational_mode and image-max-tokens
live in model-specific fields and still stay in effect until the model is reloaded.

The docs, a code comment and the API tests' notes describe the new behaviour, and
test_reasoning.think_does_not_carry_over checks it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant