Feature Request: Graceful reasoning budget termination. Avoid mid-sentence cutoff.

[CleStil]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

PR #20297 (merged ~2025-03-11) added --reasoning-budget-message, which injects
a string at the hard token cutoff to signal the model to wrap up. The injection
still happens at exactly token N with zero warning. The model gets no time to
finish its current thought before the </think> boundary.

Graceful termination before the cutoff is expected to improve reasoning
performance. See Motivation and Possible Implementation for details.

Motivation

Raw truncation of the thinking trace measurably reduces answer quality compared
to graceful termination. Muennighoff et al. (s1: Simple test-time scaling,
arXiv:2501.19393) showed that appending an end-of-thinking delimiter with
"Final Answer:" before the hard cut outperforms naïve truncation. This was
confirmed by follow-up work (arXiv:2505.05315), which states directly that
"the S1 approach performs better than directly truncating the full reasoning
trajectory, underscoring the importance of preserving the solution segment."

The current --reasoning-budget-message implementation injects a wrap-up message
at exactly token N, leaving the model zero tokens to actually act on it. This is
functionally equivalent to truncation with a cosmetic suffix — the model cannot
produce a meaningful conclusion within 0 remaining tokens.

Possible Implementation

Three approaches in increasing complexity:

Option A — message offset
Inject the budget message at budget - offset instead of budget, giving the
model offset tokens to act on the wrap-up signal. The s1 paper
(https://arxiv.org/abs/2501.19393) shows this outperforms raw truncation.
Change in the COUNTING state of reasoning-budget.cpp:

if (ctx->remaining <= ctx->warn_offset)  // was: <= 0

Files: reasoning-budget.cpp, reasoning-budget.h, common.h, arg.cpp,
sampling.cpp. Default 0 preserves current behavior.

Option B — separate budgeting
Split total budget into thinking (t) and conclusion (s) phases, t + s = budget.
Force </think> at token t, leave s tokens for a conclusion before the answer.
Follow-up work on s1 (https://arxiv.org/abs/2505.05315) shows this outperforms
the basic offset approach. Would add a second parameter, e.g.
--reasoning-budget-conclusion M.

Option C — logit biasing toward </think>
Ramp up the logit weight of </think> as the budget runs low, then force it at
the hard limit. No dependency on an injected phrase, model-agnostic. NVIDIA NIM
uses this approach with a 10% extension window for sentence completion. More
invasive (sampler changes) but cleanest long-term.

Options A and B build on the existing message injection architecture. Option C
is a parallel approach that could coexist with either.

Happy to test on ROCm/gfx1100 (Qwen3.5-27B) and contribute a draft of Option A
or B if there is interest.

[CleStil]

Update:

I tested some code:

Tested Option A (offset only) does not work because forced_tokens is atomic, model gets zero concluding tokens regardless of offset. It is just the cutoff with an extra phrase.

Tested Option B structure (two-phase) that works as intended:

Message string Final Answer:\nBased on my analysis above, works reliably for Qwen3.5-27B, Qwen3.5-35ba3b and the 9b version. Meta-comments alone about budget do not.

With sufficient concluding budget the model terminates naturally. This is exactly what we want. with forced becomes a safety net only.

Extreme test: budget=5000, inject after 200 tokens, model concluded gracefully within 4800-token window without forced termination.

"I" wrote the whole code it built and worked. The code was mainly written with an LLM thats why I will not do PR as this is against the rules. If someone is interested in my fix i am willing to share my code. If someome wants to review it can make PR but i dont want to clutter the open source space with AI code.

take away: The key insight worth posting:
The effectiveness depends entirely on the message string mimicking the model's own reasoning style rather than describing the budget constraint externally. This has implications for what the default message string should be and whether it should be model-specific. it worked great for qwen3.5 models.

[pwilkin]

We welcome all well-thought and well-designed enhancements to the reasoning sampler that do not break backwards compatibility :)

[zeel2104]

I would like to work on it. Could you assign it to me?