Skip to content

[Agents] Add Mistral Vibe CLI agent with direct API and streaming-json transcript support#137

Open
joeyspagnoli wants to merge 7 commits into
vercel-labs:mainfrom
joeyspagnoli:add-mistral-vibe-cli-agent
Open

[Agents] Add Mistral Vibe CLI agent with direct API and streaming-json transcript support#137
joeyspagnoli wants to merge 7 commits into
vercel-labs:mainfrom
joeyspagnoli:add-mistral-vibe-cli-agent

Conversation

@joeyspagnoli

@joeyspagnoli joeyspagnoli commented May 21, 2026

Copy link
Copy Markdown

Closes #90.

Adds Mistral Vibe as a new eval agent using direct API access via MISTRAL_API_KEY. The implementation follows the same pattern as the Cursor and Gemini agents, with a dedicated transcript parser that normalizes Vibe's --output streaming JSONL (OpenAI-compatible LLMMessage rows) into the canonical o11y schema.

Vibe CLI requires --prompt to enter programmatic mode (which auto-selects the auto-approve profile and bypasses tool-permission prompts that would otherwise hang in sandboxed environments), --max-turns to bound runaway agents (defaulted to 50, overridable per-experiment via agentOptions.maxTurns), and --output streaming to emit newline-delimited JSON per message for transcript parsing. We also pass --trust to silence Vibe's untrusted-workdir stderr warning and honor any project-level .vibe config — the interactive trust dialog is already skipped when --prompt is set (vibe/cli/entrypoint.py:204-206), so this is hygiene rather than a hang fix. The model is configured via the VIBE_ACTIVE_MODEL env var rather than a CLI flag — Vibe overrides any config field through VIBE_* env vars — and telemetry, OTEL exporters, and auto-update are all disabled the same way so the sandbox stays quiet.

Installation downloads the uv binary directly from GitHub releases (pinned to 0.11.15) via Node's built-in fetch and extracts with tar, rather than using upstream's curl | sh installer. This lets the same install chain work on both the Vercel sandbox (which has curl) and node:*-slim (which doesn't, and which the local Docker sandbox uses); both backends are exercised in local integration testing. uv tool install mistral-vibe then handles the Python 3.12 + Vibe install in one step. The install chain runs under set -ex so the truncated error tail identifies which stage failed if anything breaks. The default model is mistral-medium-3.5, which became Vibe's default in v2.9.1 (replacing the previous devstral-2 alias).

The parser maps Vibe's native tool names (read_file, write_file, search_replace, bash, grep, web_fetch, web_search, todo, task) to the canonical o11y tool union, extracts file paths, shell commands, and URLs into _extracted* keys, emits reasoning_content as a separate thinking event, skips harness-injected (injected: true) rows, and pairs role: 'tool' results with their originating tool calls via tool_call_id. The tool map has been verified against a captured streaming transcript from a real run; Vibe ships no glob or list_dir tool (models use bash for directory ops) and skill falls through to 'unknown' as a meta-tool with variable shape. The role: 'tool' success heuristic uses a prefix match against common error indicators (error:, failed:, traceback, etc.) so legitimate output mentioning those words mid-stream is not misclassified. Integration tests run end-to-end against the real Mistral API on the Docker sandbox.

Add Mistral Vibe as a new agent to the eval framework, enabling testing against Mistral models using direct API access via `MISTRAL_API_KEY`. The implementation follows the established direct-API pattern from existing agents (Cursor, Gemini) and integrates with the sandbox infrastructure.

Vibe uses a different invocation surface than other CLI agents in the repo. Programmatic mode is selected via `--prompt <text>`, which automatically chooses the `auto-approve` agent profile (bypassing tool permission prompts that would otherwise hang in sandboxed environments). The `--trust` flag suppresses the workdir-trust dialog required for non-interactive automation. `--max-turns` bounds runaway agents, and `--output streaming` emits one OpenAI-compatible `LLMMessage` per line for transcript parsing. The model is configured via `VIBE_ACTIVE_MODEL` rather than a CLI flag. Telemetry, OTEL exporters, and auto-update are all disabled via additional env vars so the sandbox stays quiet.

Installation uses `uv tool install mistral-vibe`, matching the Mistral Vibe upstream recommendation. `uv` auto-downloads Python 3.12 on first run (Vibe requires it). The default model is `mistral-medium-3.5`.

A new transcript parser (`parsers/mistral-vibe.ts`) normalizes Vibe's `LLMMessage` JSONL into the canonical o11y schema. It maps Vibe's native tool names (`read_file`, `write_file`, `search_replace`, `bash`, `grep`, `web_fetch`, `web_search`, `todo`, `task`) to the canonical `ToolName` union, extracts file paths, shell commands, and URLs into `_extracted*` keys, emits `reasoning_content` as a separate thinking event, skips harness-injected rows, and pairs `role: 'tool'` results with their originating tool calls via `tool_call_id`.
The original uv install chain used `curl -LsSf https://astral.sh/uv/install.sh | sh`, which fails on `node:24-slim` (the Docker sandbox image) because curl is not preinstalled and the agent does not have root access to apt-install it. Upstream's `install.sh` itself also shells out to curl/wget, so saving it to disk and running it does not help.

Fetch the uv tarball directly from GitHub releases via Node's built-in `fetch` (available in Node 24), then extract with `tar` (which is in the base image). Detect arch at runtime so the same chain works for x86_64 and aarch64 sandboxes. The rest of the install (`uv tool install mistral-vibe`, `vibe --version`) is unchanged.

This makes the agent runnable on both the Vercel sandbox (which has curl) and the local Docker sandbox (which does not), without requiring sandbox-layer changes.
Pin the uv installer to 0.11.15 instead of tracking 'latest', so an upstream
change to the uv release-tarball layout cannot silently break the Mistral
Vibe install at runtime. The pinned URL has been re-verified end-to-end on
the local Docker sandbox against a real Mistral API key.
Add a regression guard for the parser's per-id pending Map. OpenAI-compat
permits multiple tool_calls in one assistant turn and tool results can
arrive out of order, but no existing test exercised the pairing path.
The previous substring match against 'error' or 'failed' false-positived
on legitimate output — e.g. a grep hit containing the word 'error' was
marked as a failed tool result. Narrow to a prefix match against common
error indicators (error:, failed:, traceback, exception:, fatal:) after
trimming leading whitespace, and add a grep regression test.
- Switch the uv/vibe install chain to `set -ex` so the truncated error
  tail identifies which stage failed (uv download, tar extract, `uv tool
  install`, or `vibe --version`) instead of just the final exit code.
- Reword the `--trust` inline comment to match upstream: in programmatic
  mode Vibe skips the interactive trust dialog entirely
  (vibe/cli/entrypoint.py:204-206) and only emits a stderr warning, so
  `--trust` is hygiene (suppresses the warning, honors any project-level
  `.vibe` config) rather than a hang fix.
- Document the parser's tool map: Vibe ships no `glob` or `list_dir`
  tool (models use `bash` for directory ops) and `skill` falls through
  to `'unknown'` deliberately as a meta-tool with variable shape.
@benjamincanac

Copy link
Copy Markdown

@joeyspagnoli I made a similar PR #101 a while back, let me know what improvements you made to see if I should close mine!

The previous comment claimed `skill` "varies in shape per skill" —
incorrect. Vibe's `skill` tool has a fixed `{name: string}` arg shape
(vibe/core/tools/builtins/skill.py). The actual reason skill maps to
`unknown` is that the canonical `ToolName` union has no skill equivalent;
Claude Code's parser handles its `Skill` tool the same way (no entry,
falls through to unknown).

Add `skill: 'unknown'` to VIBE_TOOL_MAP explicitly, matching the Cursor
convention (cursor.ts:26 — `readLintsToolCall: 'unknown'`) of listing
known-but-unmappable tools rather than relying on implicit fallthrough.
@joeyspagnoli

Copy link
Copy Markdown
Author

hi @benjamincanac ! a few differences between #101 and this PR:

  • Default model: mistral-medium-3.5 (Vibe v2.9.1 renamed it from devstral-2; the old alias still resolves via a _migrate shim).
  • Model selection: no --model flag exists; the model is set through VIBE_ACTIVE_MODEL (the VIBE_* env override pattern documented in entrypoint.py's argparse epilog). This PR also passes --trust (added v2.9.0, silences the workdir-trust stderr warning in programmatic mode), --max-turns to bound runaway loops, and VIBE_ENABLE_{TELEMETRY,OTEL,AUTO_UPDATE}=false to quiet the sandbox.
  • Install: curl | bash doesn't work on node:24-slim (no curl, no apt root), and Mistral's own install.sh shells out to curl too. This fetches the uv tarball directly via Node's built-in fetch (pinned to 0.11.15), then uv tool install mistral-vibe. Verified end-to-end on Docker for both arm64 and amd64 against the live API.
  • Parser: Vibe's --output streaming emits OpenAI-compat LLMMessage rows (role/content/tool_calls/reasoning_content/tool_call_id), not type-discriminated events. This PR's parser handles that schema, pairs role:"tool" results with their originating call via a per-tool_call_id pending Map (parallel tool calls work), skips injected:true harness rows, and emits reasoning_content as a separate thinking event. Tool map covers 9 of the 10 active builtins; skill maps to unknown because no canonical name fits (every parser in the repo defaults unmapped tools to unknown this way). ask_user_question/exit_plan_mode are auto-disabled in programmatic mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Mistral Vibe as a supported agent harness

2 participants