VirtualMCPServer Cedar authorization rejects all requests when upstream OIDC provider issues opaque access tokens

[cjohnhanson]

Summary

When a VirtualMCPServer is configured with an embedded auth server using an upstream OIDC provider whose access tokens are opaque rather than JWTs (Google issues ya29.* opaque tokens), every Cedar authorization check fails. The vMCP gateway logs Authorization check failed for tool, skipping for every tool and returns an empty tools/list to authenticated clients.

#5002 introduced the PrimaryUpstreamProvider auto-binding for VirtualMCPServer that mirrors the working thv run path (verified in that PR against Okta, whose access tokens are JWTs). The same auto-binding causes the JWT parse step to fail unconditionally for OIDC providers that issue opaque access tokens.

Reproduction

Minimal VirtualMCPServer with Google upstream + a permit-everything Cedar policy:

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: VirtualMCPServer
metadata:
  name: gateway
spec:
  groupRef:
    name: my-mcps
  authServerConfig:
    issuer: "https://gateway.example.com"
    hmacSecretRefs:
      - name: hmac-secret
        key: hmac-key
    upstreamProviders:
      - name: google
        type: oidc
        oidcConfig:
          issuerUrl: "https://accounts.google.com"
          clientId: "..."
          clientSecretRef: { name: google-oauth, key: clientSecret }
          redirectUri: "https://gateway.example.com/oauth/callback"
          additionalAuthorizationParams:
            access_type: offline
          scopes: ["openid", "email", "profile"]
  config:
    groupRef: my-mcps
    aggregation:
      conflictResolution: prefix
      conflictResolutionConfig: { prefixFormat: "{workload}_" }
  incomingAuth:
    type: oidc
    oidcConfigRef:
      name: gateway-incoming
      audience: "https://gateway.example.com/mcp"
    authzConfig:
      type: inline
      inline:
        policies:
          - 'permit(principal, action, resource);'
  outgoingAuth:
    source: discovered

1. Apply the manifest above plus the corresponding MCPOIDCConfig and Secret resources, on v0.21.0 or later (any release containing Fix Cedar upstream-claim evaluation on VirtualMCPServer #5002).
2. Authenticate via the embedded auth server's Google OAuth flow.
3. Connect an MCP client (Claude Desktop, claude.ai, etc.) and call tools/list.

Expected behavior

The empty Cedar policy permit(principal, action, resource); permits all tools, and the client receives the aggregated tool list.

Actual behavior

Every tool is skipped with this gateway log line:

Authorization check failed for tool, skipping
  tool: <tool-name>
  error: failed to parse upstream token for provider "google":
    upstream token is not a parseable JWT: token is malformed:
    token contains an invalid number of segments

The client receives an empty tool list and cannot use the gateway.

Root cause

Verified against f9489a4b88cfdf6f984d4f665e00a01e90f13dfc (current main plus PR #4994):

1. Operator auto-binds without an opt-out. cmd/thv-operator/pkg/vmcpconfig/converter.go:201-205 unconditionally sets incoming.Authz.PrimaryUpstreamProvider to the first upstream provider's name whenever authServerConfig.upstreamProviders is non-empty. There is no CRD field exposed to override this.

2. Storage holds the ID token but the reader returns only the access token. pkg/authserver/storage/types.go defines UpstreamTokens with both AccessToken (line 61) and IDToken (line 63) fields. The OAuth callback at pkg/authserver/server/handlers/callback.go:131-140 populates both from idpTokens. pkg/auth/upstreamtoken/service.go (lines 81, 107, 122, 174) returns tokens.AccessToken only. IDToken is never plumbed into UpstreamCredential or identity.UpstreamTokens.

3. Cedar parse has no fallback. pkg/authz/authorizers/cedar/core.go:421-443 calls parseUpstreamJWTClaims(upstreamToken) whenever primaryUpstreamProvider != "", regardless of whether the policy text references upstream-prefixed claims. The function comment at line 461 explicitly notes "Returns an error if the token is not a parseable JWT (e.g. opaque token)" — but the caller returns the error directly with no fallback path.

The pkg/authserver/server/session/session.go JWT session builder already maps the upstream email/name/sub into the AS-issued JWT's claims (around lines 135-138), so falling back to identity.Claims produces the same authorization decisions for the common case where policies reference the standard OIDC claims.

Proposed fix

Smallest defensible change: in resolveClaims, when parseUpstreamJWTClaims returns the "not a parseable JWT" error, log a WARN and fall back to jwt.MapClaims(identity.Claims). Preserves Okta and other JWT-access-token providers (no behavior change) while letting opaque-access-token providers like Google reuse the AS-issued token's claims.

A complementary operator change — only auto-set PrimaryUpstreamProvider when the user hasn't asked for the explicit fallback — would expose the documented escape hatch via a CRD field. Not required for the immediate bug fix but worth pairing.

Workarounds today

• incomingAuth.authzConfig.type: "none" disables Cedar entirely.
• Removing authServerConfig.upstreamProviders and pointing incomingAuth.oidcConfigRef directly at Google as the OIDC issuer keeps Cedar functional but loses the embedded-AS features (token persistence, server-side token exchange to backends).

Environment

• ToolHive operator + vmcp built from f9489a4b88cfdf6f984d4f665e00a01e90f13dfc (current main + PR Allow standalone Redis in auth server storage #4994).
• Confirmed against v0.26.1 stock images carrying the same converter and Cedar code.

Intent to contribute

Planning to submit a PR with the Cedar fallback plus a unit test covering the opaque-upstream-token path. Happy to also pair the operator opt-out change if that's of interest.

[cjohnhanson]

One more piece of context I missed before filing: pkg/authz/authorizers/cedar/core_test.go already has a test case "upstream_token_opaque_not_parseable" (in TestAuthorizeWithJWTClaims_UpstreamProvider, around line 1297) that explicitly asserts the error path for opaque upstream tokens. So the current behavior is intentional and tested, not an oversight in #5002.

That re-frames the issue. Two coherent design positions:

1. The test is right; the operator auto-binding is wrong. convertIncomingAuth shouldn't unilaterally enable upstream-claim mode for every VirtualMCPServer with upstream providers — opaque-access-token providers exist and the user should opt in. Fix is to add a CRD field on AuthzConfig (e.g. primaryUpstreamProvider as *string) and only auto-derive when the user hasn't set it. Existing test stays green.

2. The behavior is wrong; the test should change. When upstream is opaque, fall back to identity claims (which already carry upstream email/sub/name from the AS-issued token, so most policies still evaluate as intended). Test gets updated to assert the fallback. Trade-off: users who relied on the "deny if upstream isn't a JWT" semantic break, even if just because they had no other config knob to express that intent.

Either fix unblocks Google. (1) is more conservative and probably the right architectural choice; (2) is smaller-surface and helps anyone with this config without requiring a new opt-in. Happy to PR either — let me know which you'd prefer, or I can open both for comparison.

[danbarr]

Confirming this defect also reproduces on MCPServer (not just VirtualMCPServer), via the same injectUpstreamProviderIfNeeded path.

Trigger: any MCPServer with both authzConfig and an embedded auth server whose upstream issues opaque access tokens (GitHub PATs gho_*/ghp_*, Google ya29.*, etc.). Empty tools/list regardless of policy content (verified with permit(principal, action, resource);).

Smoking gun in the toolhive proxy logs:

{"level":"WARN","msg":"Authorization check failed for tool, skipping","tool":"update_pull_request","error":"failed to parse upstream token for provider \"github\": upstream token is not a parseable JWT: token is malformed: token contains an invalid number of segments"}

MCPServer code path (mirrors the VirtualMCPServer one):

• pkg/runner/middleware.go:171-185 — PopulateMiddlewareConfigs unconditionally calls injectUpstreamProviderIfNeeded(config.AuthzConfig, config.EmbeddedAuthServerConfig) whenever AuthzConfig != nil.
• pkg/runner/middleware.go:379-397 — the helper sets Cedar.PrimaryUpstreamProvider whenever embeddedCfg != nil. No opt-out, no CRD knob on MCPServer.spec.authzConfig either.
• cmd/thv-operator/controllers/mcpserver_runconfig.go:292 calls PopulateMiddlewareConfigs, so the operator path inherits the same auto-binding.

#5147 fixes this for MCPServer too, since resolveClaims is shared. Worth widening the PR description / `Closes` line to call out both surfaces.