-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Problem
When running evals with the copilot target, an MCP server with an expired OAuth token triggers the OAuth flow. If the OAuth flow is not completed (e.g., in non-interactive/automated eval runs), copilot CLI hangs waiting for auth, blocking the eval from completing.
Note: MCP servers do work in copilot CLI non-interactive mode (upstream issues were resolved). The problem is specifically when an OAuth token has expired and the re-auth flow requires user interaction.
Observed Behavior
During a multi-target eval run (vscode + copilot), copilot failed to complete because an MCP server's OAuth token had expired, triggering an interactive auth prompt that blocked execution.
Expected Behavior
The copilot provider should either:
- Detect expired MCP OAuth tokens and skip those MCP servers gracefully
- Timeout and continue without the MCP server if auth cannot be completed
- Warn the user about expired tokens before starting the eval
Workaround
Remove MCP server configurations with expired OAuth tokens from copilot before running evals, or re-authenticate manually before starting the eval run.
Upstream References
- copilot-cli#1797: Use OAuth refresh_token grant to silently refresh expired MCP tokens on session start — When an MCP OAuth token expires, the CLI doesn't use the refresh token to silently renew it. Instead it requires manual
/mcpre-auth via browser, which blocks non-interactive execution. - copilot-cli#1491: OAuth token refresh fails with invalid redirect_uri for enterprise MCP servers — Even when token refresh is attempted, it can fail because the CLI uses random ephemeral ports for the OAuth callback, which don't match the port registered during initial auth. This forces re-authentication via browser on every token expiration.
A new upstream issue should be filed specifically for the non-interactive blocking behavior (expired OAuth token hangs the CLI instead of failing gracefully).
Context
Found during #458 — testing multi-repo workspace example with both vscode and copilot targets.