`Error: No tokenizer found for model` with Ollama

[luochen1990]

Describe the bug

Trying to use ollama but failed with Error: No tokenizer found for model, tried to change the model but still same error.

To Reproduce

[openai]
api_base = "http://127.0.0.1:11434/v1"
api_key = ""
model = "llama3.1:latest"
retries = 2
proxy = ""

1. Run ollama server locally
2. Exec ollama run llama3.1:latest
3. Try some talk to Confirm that ollama is running without issue
4. Edit ~/.config/gptcommit/config.toml and add above config
5. git add something and Execute git commit
6. See error

Expected behavior

Works correctly and without error

Screenshots
If applicable, add screenshots to help explain your problem.
Run your command with the RUST_LOG=trace environment variable for the best support

System Info (please complete the following information):

• OS: linux (NixOS unstable)
• Version: gptcommit v0.5.16

[assafmo]

Happens to me to with groq:

[openai]
api_base = "https://api.groq.com/openai/v1"
api_key = "XXX"
model = "llama-3.2-90b-text-preview"
retries = 2
proxy = ""

[scylamb]

Happens to me to with gpt-4o-mini

STDERR: Error: No tokenizer found for model gpt-4o-mini

[rmnilin]

This error occurs here

gptcommit/src/llms/openai.rs

Line 101 in 94eeb2f

let prompt_token_limit = get_completion_max_tokens(&self.model, prompt)?;

because this library relies on https://github.com/zurawiki/tiktoken-rs to count and limit prompt tokens. Both libraries are intended to be used exclusively with OpenAI models:

• Add Bedrock, Ollama, Claude2, Cohere, Replicate [100+ LLMs] - using LiteLLM #246
• https://github.com/zurawiki/tiktoken-rs/blob/088ff6d51629acea1624298151df148f328d6d0c/tiktoken-rs/src/tokenizer.rs#L33.

For ollama users, there's a workaround that involves creating an alias for a target model. Here's how you can do it:

$ ollama cp qwen2.5-coder:7b gpt-4--gptcommit-workaround-alias--qwen2.5-coder:7b

# gptcommit/config.toml
[openai]
api_base = "http://localhost:11434/v1"
api_key = ""
model = "gpt-4--gptcommit-workaround-alias--qwen2.5-coder:7b"
retries = 2
proxy = ""

To remove the alias when you no longer need it, use the following command:

$ ollama rm gpt-4--gptcommit-workaround-alias--qwen2.5-coder:7b

Only alias will be removed, not the original model.

Also note that you can use any name or prefix from the zurawiki/tiktoken-rs enums linked above, but there are different logic depending on the model prefix, see

gptcommit/src/llms/openai.rs

Line 95 in 94eeb2f

pub(crate) fn should_use_chat_completion(model: &str) -> bool {

[hellopenggao]

Thanks, @rmnilin ! Your analysis inspired me. I solved this issue with a single command.

I'm using Hugging Face + text-generation-webui. So, I created a symbolic link in the model directory of text-generation-webui:

ln -s ~/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-14B-Instruct-GPTQ-Int4/snapshots/d435273c87c9fd984eb9d27ae89d4e12a4676052/ gpt-4--gptcommit-workaround-alias--qwen2.5-coder:14b

I used the name you recommended for the symlink, and now the loaded model can be properly recognized in gptcommit.

[ngn999]

model = "grok-2-latest" same error.

[medopaw]

@zurawiki Could we add a configuration setting to enforce the use of chat completion? For example, enforce_chat_completion.