Repository navigation
Conversation
…request
A setting one request gave, such as think or temperature, stayed on the shared engine and
applied to every later request that left it out, from any client: after one request with
think: true, requests without it still got thinking. The docs warned about this ("Settings
stick").
The engine remembers its sampler config and whether thinking is on once the model is loaded,
and the server puts them back before applying a request's settings, on the chat endpoints and
on /api/generate. As the sampler is recreated, every request also starts with an empty penalty
window. Qwen3.5-Omni's set_sampler override did the same as the base one, so it is gone.
The reasoning_effort levels of GPT-OSS and Hy-Memory, conversational_mode and image-max-tokens
live in model-specific fields and still stay in effect until the model is reloaded.
The docs, a code comment and the API tests' notes describe the new behaviour, and
test_reasoning.think_does_not_carry_over checks it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The server applies a request's settings (
think,reasoning_effort,temperature,top_p,top_k,min_p, the penalties) to the shared engine and never resets them. A setting one request gives stays in place for every later request that leaves it out, from any client. For example, after one request with"think": true, requests withoutthinkstill render<|think|>and the model keeps reasoning. The docs warned about this ("Settings stick").The engine now remembers its sampler config and whether thinking is on once the model is loaded, and the server puts them back before applying each request's settings, on the chat endpoints and on
/api/generate. A setting a request leaves out falls back to the model's default. Because the sampler is recreated, every request also starts with an empty penalty window.The docs, a code comment and the API tests' notes now describe the new behaviour, and
test_reasoning.think_does_not_carry_overchecks it.Qwen3_5_Omni::set_samplerdid the same asAutoModel::set_sampler, so it is removed; otherwise Qwen3.5-Omni would skip remembering its sampler config.The
reasoning_effortlevels of GPT-OSS and Hy-Memory,conversational_modeandimage-max-tokensare stored in model-specific fields and still stay in effect until the model is reloaded.I noticed this while reading the prompts flm renders, with a small debug branch that logs every prompt and raw output to a file (
log-prompt,FLM_LOG_PROMPT=<file>): a request withoutthinkstill had<|think|>in its prompt.