Tool support

[simonw]

Some llama-server models support tools. https://llm.datasette.io/en/stable/changelog.html#v0-26

[simonw]

Got this working, had to do --no-stream though:

% llm chat -m llama-server  -T llm_version -T llm_time --td --no-stream          
Chatting with llama-server
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> what time is it?

Tool call: llm_time({})
  {
    "utc_time": "2025-05-28 06:02:45 UTC",
    "utc_time_iso": "2025-05-28T06:02:45.119726+00:00",
    "local_timezone": "PDT",
    "local_time": "2025-05-27 23:02:45",
    "timezone_offset": "UTC-7:00",
    "is_dst": true
  }

It is currently 2025-05-27 23:02:45 PDT. Daylight saving time is in effect.

I was running this in another window:

llama-server --jinja -hf unsloth/gemma-3-4b-it-GGUF:Q4_K_XL

[simonw]

Without --no-stream I get this:

openai.InternalServerError: Error code: 500 - {'error': {'code': 500, 'message': 'Cannot use tools with stream', 'type': 'server_error'}}

[simonw]

I'm going to create a llama-server-tools model ID that has streaming disabled and tools enabled.

[simonw]

OK this works!

llm -m llama-server-tools -T llm_time 'time?' --td

Tool call: llm_time({})
  {
    "utc_time": "2025-05-28 06:07:57 UTC",
    "utc_time_iso": "2025-05-28T06:07:57.606470+00:00",
    "local_timezone": "PDT",
    "local_time": "2025-05-27 23:07:57",
    "timezone_offset": "UTC-7:00",
    "is_dst": true
  }

The current time is 2025-05-27 23:07:57 PDT.

[simonw]

Looks like this landed three days ago:

• server: streaming of tool calls and thoughts when --jinja is on ggml-org/llama.cpp#12379

So streaming tools might be possible pretty soon!