Skip to content

feat(gemini-adapter): prefer HTTP API when GEMINI_API_KEY is set#1365

Closed
abigail-atheryon wants to merge 2 commits intogarrytan:mainfrom
abigail-atheryon:feat/gemini-http-api
Closed

feat(gemini-adapter): prefer HTTP API when GEMINI_API_KEY is set#1365
abigail-atheryon wants to merge 2 commits intogarrytan:mainfrom
abigail-atheryon:feat/gemini-http-api

Conversation

@abigail-atheryon
Copy link
Copy Markdown

@abigail-atheryon abigail-atheryon commented May 7, 2026

Summary

The OAuth tier hits hard daily quotas on gemini-2.5-pro after a handful of calls (verified empirically — repeated /benchmark-models runs trip "You have exhausted your capacity on this model"). The CLI's stream-json output also reports 0/0 tokens for OAuth runs, so cost numbers in the benchmark table were always $0.0000 — even when real billing was happening on paid accounts.

This patch prefers the Generative Language HTTP API directly when GEMINI_API_KEY (or GOOGLE_API_KEY) is set:

  • Paid quotas (no daily cap surprises mid-benchmark)
  • Full usage data, including thoughtsTokenCount for reasoning models — folded into output token count so cost estimation matches the real bill
  • No CLI dependency on the API-key path

CLI fallback retained for free-OAuth users with no API key, so /benchmark-models still works out of the box on a fresh install.

Stacked on

Stacks on top of #1362 (the OAuth path-detection fix). Targets main; will rebase cleanly once #1362 lands.

Auth precedence

  1. GEMINI_API_KEY → HTTP path (preferred)
  2. GOOGLE_API_KEY → HTTP path
  3. ~/.gemini/oauth_creds.json or ~/.config/gemini/ → CLI path

Verification

Before (OAuth, gemini-2.5-pro):
```
gemini-2.5-pro 275.6s 0→0 $0.0000 ERROR rate_limit
```

After (HTTP, GEMINI_API_KEY set):
```
gemini-2.5-pro 12.7s 23→1294 $0.0065 OK
```

Same prompt, same gstack-model-benchmark invocation. Only GEMINI_API_KEY was added.

Test plan

  • HTTP path returns real tokens + cost for gemini-2.5-pro and gemini-2.5-flash
  • CLI fallback still works when no API key is set (verified with key unset)
  • OAuth-only users see unchanged behavior
  • Reasoning ("thoughts") tokens correctly attributed to output for cost
  • Maintainer to verify on a Vertex/non-personal account if relevant

Notes

The HTTP rewrite is roughly the same shape as the existing GPT adapter (CLI shell-out) but uses fetch() for the API-key path. No new deps, no test infra changes.

🤖 Generated with Claude Code


View in Codesmith
Need help on this PR? Tag @codesmith with what you need.

  • Let Codesmith autofix CI failures and bot reviews

Abigail Atheryon added 2 commits May 8, 2026 07:34
gemini-cli >=0.30 stores OAuth credentials at ~/.gemini/oauth_creds.json
instead of the legacy ~/.config/gemini/ directory. The benchmark adapter's
availability check now succeeds for users on recent gemini-cli releases
who have authenticated via interactive login.

Both paths are accepted so users on older versions still work.
The OAuth tier hits hard daily quotas on gemini-2.5-pro after a handful
of calls, and the CLI's stream-json output reports 0/0 tokens for
OAuth runs (so cost numbers were always $0.0000). Switch to the
Generative Language HTTP API directly when GEMINI_API_KEY (or
GOOGLE_API_KEY) is set — paid quotas, full usage data including
reasoning ("thoughts") tokens which are folded into output for accurate
cost estimation.

CLI fallback retained for free-OAuth users with no API key, so /benchmark
still works out of the box on a fresh install.

Auth precedence:
  1. GEMINI_API_KEY  → HTTP path (preferred)
  2. GOOGLE_API_KEY  → HTTP path
  3. ~/.gemini/oauth_creds.json or ~/.config/gemini/  → CLI path
@garrytan
Copy link
Copy Markdown
Owner

Thanks @abigail-atheryon — closing as deferred. Gemini HTTP API preference is a substantial behavioral change to the gemini adapter (112 lines). Companion #1362 (oauth path detect) shipped in a prior wave; the HTTP API switch warrants focused review on its own. Happy to revisit.

@garrytan garrytan closed this May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants