[Agents] Add Mistral Vibe CLI agent with direct API and streaming-json transcript support#137
Open
joeyspagnoli wants to merge 7 commits into
Open
[Agents] Add Mistral Vibe CLI agent with direct API and streaming-json transcript support#137joeyspagnoli wants to merge 7 commits into
joeyspagnoli wants to merge 7 commits into
Conversation
Add Mistral Vibe as a new agent to the eval framework, enabling testing against Mistral models using direct API access via `MISTRAL_API_KEY`. The implementation follows the established direct-API pattern from existing agents (Cursor, Gemini) and integrates with the sandbox infrastructure. Vibe uses a different invocation surface than other CLI agents in the repo. Programmatic mode is selected via `--prompt <text>`, which automatically chooses the `auto-approve` agent profile (bypassing tool permission prompts that would otherwise hang in sandboxed environments). The `--trust` flag suppresses the workdir-trust dialog required for non-interactive automation. `--max-turns` bounds runaway agents, and `--output streaming` emits one OpenAI-compatible `LLMMessage` per line for transcript parsing. The model is configured via `VIBE_ACTIVE_MODEL` rather than a CLI flag. Telemetry, OTEL exporters, and auto-update are all disabled via additional env vars so the sandbox stays quiet. Installation uses `uv tool install mistral-vibe`, matching the Mistral Vibe upstream recommendation. `uv` auto-downloads Python 3.12 on first run (Vibe requires it). The default model is `mistral-medium-3.5`. A new transcript parser (`parsers/mistral-vibe.ts`) normalizes Vibe's `LLMMessage` JSONL into the canonical o11y schema. It maps Vibe's native tool names (`read_file`, `write_file`, `search_replace`, `bash`, `grep`, `web_fetch`, `web_search`, `todo`, `task`) to the canonical `ToolName` union, extracts file paths, shell commands, and URLs into `_extracted*` keys, emits `reasoning_content` as a separate thinking event, skips harness-injected rows, and pairs `role: 'tool'` results with their originating tool calls via `tool_call_id`.
The original uv install chain used `curl -LsSf https://astral.sh/uv/install.sh | sh`, which fails on `node:24-slim` (the Docker sandbox image) because curl is not preinstalled and the agent does not have root access to apt-install it. Upstream's `install.sh` itself also shells out to curl/wget, so saving it to disk and running it does not help. Fetch the uv tarball directly from GitHub releases via Node's built-in `fetch` (available in Node 24), then extract with `tar` (which is in the base image). Detect arch at runtime so the same chain works for x86_64 and aarch64 sandboxes. The rest of the install (`uv tool install mistral-vibe`, `vibe --version`) is unchanged. This makes the agent runnable on both the Vercel sandbox (which has curl) and the local Docker sandbox (which does not), without requiring sandbox-layer changes.
Pin the uv installer to 0.11.15 instead of tracking 'latest', so an upstream change to the uv release-tarball layout cannot silently break the Mistral Vibe install at runtime. The pinned URL has been re-verified end-to-end on the local Docker sandbox against a real Mistral API key.
Add a regression guard for the parser's per-id pending Map. OpenAI-compat permits multiple tool_calls in one assistant turn and tool results can arrive out of order, but no existing test exercised the pairing path.
The previous substring match against 'error' or 'failed' false-positived on legitimate output — e.g. a grep hit containing the word 'error' was marked as a failed tool result. Narrow to a prefix match against common error indicators (error:, failed:, traceback, exception:, fatal:) after trimming leading whitespace, and add a grep regression test.
- Switch the uv/vibe install chain to `set -ex` so the truncated error tail identifies which stage failed (uv download, tar extract, `uv tool install`, or `vibe --version`) instead of just the final exit code. - Reword the `--trust` inline comment to match upstream: in programmatic mode Vibe skips the interactive trust dialog entirely (vibe/cli/entrypoint.py:204-206) and only emits a stderr warning, so `--trust` is hygiene (suppresses the warning, honors any project-level `.vibe` config) rather than a hang fix. - Document the parser's tool map: Vibe ships no `glob` or `list_dir` tool (models use `bash` for directory ops) and `skill` falls through to `'unknown'` deliberately as a meta-tool with variable shape.
|
@joeyspagnoli I made a similar PR #101 a while back, let me know what improvements you made to see if I should close mine! |
The previous comment claimed `skill` "varies in shape per skill" —
incorrect. Vibe's `skill` tool has a fixed `{name: string}` arg shape
(vibe/core/tools/builtins/skill.py). The actual reason skill maps to
`unknown` is that the canonical `ToolName` union has no skill equivalent;
Claude Code's parser handles its `Skill` tool the same way (no entry,
falls through to unknown).
Add `skill: 'unknown'` to VIBE_TOOL_MAP explicitly, matching the Cursor
convention (cursor.ts:26 — `readLintsToolCall: 'unknown'`) of listing
known-but-unmappable tools rather than relying on implicit fallthrough.
Author
|
hi @benjamincanac ! a few differences between #101 and this PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #90.
Adds Mistral Vibe as a new eval agent using direct API access via
MISTRAL_API_KEY. The implementation follows the same pattern as the Cursor and Gemini agents, with a dedicated transcript parser that normalizes Vibe's--output streamingJSONL (OpenAI-compatibleLLMMessagerows) into the canonical o11y schema.Vibe CLI requires
--promptto enter programmatic mode (which auto-selects theauto-approveprofile and bypasses tool-permission prompts that would otherwise hang in sandboxed environments),--max-turnsto bound runaway agents (defaulted to 50, overridable per-experiment viaagentOptions.maxTurns), and--output streamingto emit newline-delimited JSON per message for transcript parsing. We also pass--trustto silence Vibe's untrusted-workdir stderr warning and honor any project-level.vibeconfig — the interactive trust dialog is already skipped when--promptis set (vibe/cli/entrypoint.py:204-206), so this is hygiene rather than a hang fix. The model is configured via theVIBE_ACTIVE_MODELenv var rather than a CLI flag — Vibe overrides any config field throughVIBE_*env vars — and telemetry, OTEL exporters, and auto-update are all disabled the same way so the sandbox stays quiet.Installation downloads the
uvbinary directly from GitHub releases (pinned to0.11.15) via Node's built-infetchand extracts withtar, rather than using upstream'scurl | shinstaller. This lets the same install chain work on both the Vercel sandbox (which has curl) andnode:*-slim(which doesn't, and which the local Docker sandbox uses); both backends are exercised in local integration testing.uv tool install mistral-vibethen handles the Python 3.12 + Vibe install in one step. The install chain runs underset -exso the truncated error tail identifies which stage failed if anything breaks. The default model ismistral-medium-3.5, which became Vibe's default in v2.9.1 (replacing the previousdevstral-2alias).The parser maps Vibe's native tool names (
read_file,write_file,search_replace,bash,grep,web_fetch,web_search,todo,task) to the canonical o11y tool union, extracts file paths, shell commands, and URLs into_extracted*keys, emitsreasoning_contentas a separatethinkingevent, skips harness-injected (injected: true) rows, and pairsrole: 'tool'results with their originating tool calls viatool_call_id. The tool map has been verified against a captured streaming transcript from a real run; Vibe ships nogloborlist_dirtool (models usebashfor directory ops) andskillfalls through to'unknown'as a meta-tool with variable shape. Therole: 'tool'success heuristic uses a prefix match against common error indicators (error:,failed:,traceback, etc.) so legitimate output mentioning those words mid-stream is not misclassified. Integration tests run end-to-end against the real Mistral API on the Docker sandbox.