[BUG]: InteractiveExecutor when using MTMD with a limited context size, a NoKvSlot error occurs

[aropb]

Description

InteractiveExecutor when using MTMD with a limited context size, a NoKvSlot error occurs

InferInternal()
...

LLamaSharp/LLama/LLamaInteractExecutor.cs

Line 251 in bcc4bc1

var result = await Context.DecodeAsync(_embeds, LLamaSeqId.Zero, batch, _pastTokensCount);

Variable for example:
Context.ContextSize = 1792
_pastTokensCount = 551
_embeds.Count = 1
_session_tokens.Count = 550
_embed_inps.Count = 213

The error occurs only when adding an image and using MTMD.

I've been trying to figure out what's going on for hours now. The variables show that there is still a free context for tokens. There is no shift. And an error occurs. Obviously, the context is overflowing, but what and why are the current variables (_pastTokensCount, _embeds.Count) it doesn't reflect I don't understand.

Environment & Configuration

• Operating system: Windows
• .NET runtime version: 10.0.4
• LLamaSharp version: 0.26.0
• CUDA version (if you are using cuda backend): 12.9
• CPU & GPU device: RTX 5090

[aropb]

@SignalRT
Please help me figure out the error. It's easy to check, you need to create a context less than you need for the prompt, image, and response.

[aropb]

That's what I don't understand, why is that?

_mtmdWeights.NativeHandle.CountTokens(chunks) => 1455
_mtmdWeights.EvaluateChunks(...) => new pastTokensCount = 213

            foreach (SafeMtmdInputChunk chunk in chunks.Enumerate())
            {
                using SafeMtmdInputChunk scopedChunk = chunk;

                if (scopedChunk.Type == SafeMtmdInputChunk.SafeMtmdInputChunkType.Text)
                {
                    foreach (var token in scopedChunk.GetTextTokensSpan())
                    {
                        tokens.Add(token);
                    }
                }
            }

tokens.Count = 167

Why are all the values different, especially (CountTokens() and pastTokensCount) with such a big difference?

[aropb]

@SignalRT
I kind of figured out what's going on, here's an example of how to count tokens:

https://github.com/ggml-org/llama.cpp/blob/d0b79aaa2f6e7b7d3c26b1845b43cef158697540/tools/server/server-common.cpp#L342

Then, in my case, tokens contains all the necessary tokens and a context shift occurs on time.
in other words, the number of tokens in the list should be equal to _mtmdWeights.NativeHandle.CountTokens(chunks)!

But it is important that the pastTokensCount will be smaller!

[SignalRT]

@aropb I will be reviewing this an finishing the PR that I working on these days.

[aropb]

            foreach (SafeMtmdInputChunk chunk in chunks.Enumerate())
            {
                using SafeMtmdInputChunk scopedChunk = chunk;

                if (scopedChunk.Type == SafeMtmdInputChunk.SafeMtmdInputChunkType.Text)
                {
                    foreach (var token in scopedChunk.GetTextTokensSpan())
                    {
                        tokens.Add(token);
                    }
                }
                else
                {
                    tokens.AddRange(Enumerable.Repeat(fillerToken, (int)scopedChunk.NTokens));
                }
            }

the context shift should be done by tokens.Count(), not n_past

[aropb]

But why for mtmd, after EvaluateChunks(), n_past is not equal to the number of tokens is not clear, can someone explain?

And maybe it's possible to get all tokens from chunks, not just text ones?

[SignalRT]

@aropb In this PR #1316 I make several changes to the mtmd management. I also try to reproduce this limitations:

• No multimodal session persistence.
• No multimodal context shifting when the prompt overflows the KV cache.
• No safe reuse of text-only session-prefix caching for multimodal prompts.
• Multimodal input must be prepared explicitly through MTMD media embeddings instead of ordinary text tokenization.

If you have time to test it, that would be very helpful.

[aropb]

Unfortunately, I won't be able to check the llama.web and executors of LLamaSharp.

SafeMtmdModelHandle.cs and MtmdWeights.cs
I'm sure it needs to be removed: _pendingMedia and lock (_syncRoot).

This is wrong. Only Executors should have Embedds (media).
There is one instance of the model for the entire web application. I don't understand why embed should be added to the model.

Please, explain this:
"Multimodal input must be prepared explicitly through MTMD media embeddings instead of ordinary text tokenization."

[SignalRT]

_pendingMedia and lock (_syncRoot) were added because MtmdWeights is currently shared at model scope, while the old MTMD API exposed a mutable “pending media” queue on the model handle itself. In a web app where one model instance is shared across sessions, that queue is unsafe: two sessions could enqueue media on the same model and interfere with each other. The lock was only a defensive guard around that shared mutable state.

Conceptually, I agree with your point: media ownership should live at executor/session level, not at model level. That is exactly why the new standalone APIs (LoadMediaStandalone(...)) were introduced. They let a session prepare its own SafeMtmdEmbed and pass it explicitly to tokenization, without touching the shared model queue. In that design, _pendingMedia is just a legacy compatibility path. If we fully migrate all callers to explicit per-executor embeds, then _pendingMedia should be removed, and the model-level lock can be reduced or removed as well.

About this sentence:

“Multimodal input must be prepared explicitly through MTMD media embeddings instead of ordinary text tokenization.”

What it means is:

Text tokenization only handles text. MTMD media has to be prepared separately and attached explicitly, and that attachment belongs logically to the executor/session, not to the shared model instance.

[aropb]

I still don't understand :) What's the point? Moreover, you don't even shift the context.