Skip to content

bug: copilot CLI MCP auth prompt blocks eval execution #461

@christso

Description

@christso

Problem

When running evals with the copilot target, an MCP server with an expired OAuth token triggers the OAuth flow. If the OAuth flow is not completed (e.g., in non-interactive/automated eval runs), copilot CLI hangs waiting for auth, blocking the eval from completing.

Note: MCP servers do work in copilot CLI non-interactive mode (upstream issues were resolved). The problem is specifically when an OAuth token has expired and the re-auth flow requires user interaction.

Observed Behavior

During a multi-target eval run (vscode + copilot), copilot failed to complete because an MCP server's OAuth token had expired, triggering an interactive auth prompt that blocked execution.

Expected Behavior

The copilot provider should either:

  • Detect expired MCP OAuth tokens and skip those MCP servers gracefully
  • Timeout and continue without the MCP server if auth cannot be completed
  • Warn the user about expired tokens before starting the eval

Workaround

Remove MCP server configurations with expired OAuth tokens from copilot before running evals, or re-authenticate manually before starting the eval run.

Upstream References

A new upstream issue should be filed specifically for the non-interactive blocking behavior (expired OAuth token hangs the CLI instead of failing gracefully).

Context

Found during #458 — testing multi-repo workspace example with both vscode and copilot targets.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions