Skip to content

feat(llmhubs): support tool calling and structured output in the Gemi…#45

Open
xinkuleee wants to merge 3 commits into
microsoft:mainfrom
xinkuleee:feat/gemini-tool-calling
Open

feat(llmhubs): support tool calling and structured output in the Gemi…#45
xinkuleee wants to merge 3 commits into
microsoft:mainfrom
xinkuleee:feat/gemini-tool-calling

Conversation

@xinkuleee

Copy link
Copy Markdown

Summary

Adds Google Gemini function/tool calling and JSON-schema structured output to the
LLMHub Gemini adapter, so Gemini models can drive the chat agent (tools) and
structured routing (intent check).

  • Tool calling: map internal (OpenAI-shaped) tools / function_call /
    function_result to Gemini functionDeclarations / functionCall /
    functionResponse (role mapping assistant→model / tool→user, and
    call_id↔function-name correlation since Gemini responses carry no call id);
    parse Gemini functionCall back into function_call outputs.
  • Schema sanitizer: rewrite tool/response JSON Schema into Gemini's
    OpenAPI-subset — inline $ref/$defs, drop additionalProperties/$schema,
    flatten single-element allOf, and convert Optional anyOf + {"type":"null"}
    to nullable.
  • Structured output: map response_format.type=json_schema to Gemini
    responseMimeType + responseSchema (fixes chat intent-check routing on Gemini).
  • Capability flags: advertise supports_tools / supports_structured_output
    for the Gemini provider (core io_profile derivation + backend equivalent).
  • Tests: unit coverage for request building, input/response mapping, the
    sanitizer (refs/defs/additionalProperties/allOf/anyOf-null/recursion), and
    structured output.

Validation

  • make precommit-run
  • cd backend && go test ./...
  • cd core && uv run pytest
  • Frontend build/lint, if working from a frontend source checkout
  • Not run; explain below

Frontend is untouched, so its build/lint does not apply.

Checklist

  • Linked relevant issues or explained why there is no issue.
  • Updated docs and examples where behavior changed.
  • Updated CHANGELOG.md under ## [Unreleased] for user-facing changes.
  • Regenerated and committed generated files after proto, Wire, or OpenAPI changes.
  • Confirmed no secrets, tokens, credentials, or sensitive logs are included.

Notes for reviewers

  • Streaming is not implemented for Gemini yet — it falls back to the base
    non-streaming path (generate_stream yields a single chunk). Tool calling works,
    but without incremental token streaming; a follow-up can add
    streamGenerateContent?alt=sse.
  • The schema sanitizer targets the keywords Gemini currently rejects; verified
    against the full builtin tool set and the structured intent-check schema.
  • No secrets/config included: the local Gemini model YAML (with API key) and the
    local default-model switch are gitignored and intentionally excluded from this PR.

Comment thread core/app/llmhubs/adapters/gemini.py Outdated
Comment thread core/app/llmhubs/adapters/gemini.py Outdated
Comment thread core/app/llmhubs/adapters/gemini.py
Comment thread core/app/llmhubs/adapters/gemini.py
Comment thread core/app/llmhubs/adapters/gemini.py
…arts

Sanitize response/tool schemas against an explicit allowlist of Gemini Schema fields: convert single-value const to enum and drop unsupported format values and constraints (exclusiveMinimum/Maximum, multipleOf, ...) instead of forwarding them and failing the whole request. Also preserve and replay the original signed Part for text outputs (not only function calls) so a Gemini 3 thoughtSignature on a text Part survives the parse -> Content -> next-turn round trip.
@jasonchen922

Copy link
Copy Markdown
Contributor

the new allowlist filters schema field names, but it does not validate the value shape required by Gemini's OpenAPI Schema.

In particular, Schema.enum accepts strings only, while the current const conversion emits the original JSON value unchanged. The official google-genai types.Schema validator rejects all of these sanitized Pydantic schemas:

  • Literal[3] -> {"type": "integer", "enum": [3]}
  • Literal[True] -> {"type": "boolean", "enum": [true]}
  • Literal[None] -> an enum containing null
  • IntEnum -> an integer-valued enum

The newly added numeric-const test therefore asserts an output that Gemini's Schema contract rejects. The same value-shape issue also applies to type: a valid JSON Schema such as {"type": ["string", "null"]} currently passes through unchanged, but Gemini's Schema.type accepts only one enum value.

Please validate and normalize field values in addition to allowlisting keys. Non-string enums should either be represented using supported constraints, deliberately dropped/rejected, or sent through the JSON-Schema-specific API fields (responseJsonSchema / parametersJsonSchema). Nullable type arrays should be converted to a scalar type plus nullable, where representable. Please add contract tests covering numeric/bool/null literals, IntEnum, and type arrays.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants