Skip to content

FORGE-EVAL-2: OTel exporter to AIP eval collector via OIDC token #190

Description

@initializ-mk

Context

Forge agents already emit OpenInference-annotated OTel spans (via the framework's tracing). The destination is currently configurable per-deployment. Wire it explicitly to AIP's eval collector with OIDC authentication so agents-as-deployed automatically stream traces.

Scope

Configuration

New `forge.yaml` block (or env-driven — design choice; recommend env-driven so it can be deployment-time, not build-time):

```
FORGE_OTEL_ENDPOINT=https://eval.aip.initializ.ai:4317
FORGE_OTEL_PROTOCOL=grpc
FORGE_OTEL_AUTH=oidc # uses the same OIDC token the agent uses for A2A
```

When `FORGE_OTEL_AUTH=oidc`:

  • The OpenTelemetry exporter's gRPC metadata includes `Authorization: Bearer `
  • Token comes from Forge's existing OIDC token source (same one used for A2A peer auth)
  • Token refresh handled by the existing token cache; exporter retries on auth failure

Span attribute tenancy stamping

Forge already stamps `FORGE_ORG_ID` / `FORGE_WORKSPACE_ID` / `FORGE_AGENT_ID` on resource attributes. Verify these become OpenInference-conformant resource attributes that the eval collector (initializ/eval#3) extracts and cross-checks against the token's tenancy claims.

Failure handling

  • If `FORGE_OTEL_ENDPOINT` is unset → no export (current behaviour preserved)
  • If endpoint is set but unreachable → buffer in memory up to N spans, drop with metric (don't block agent)
  • Exporter respects the same fire-and-forget posture as the audit emission — never block the agent

Why OIDC reuse

Forge already has the OIDC machinery for A2A auth. The eval collector (initializ/eval#4) validates against the same Okta JWKS. Same trust chain, no new credentials to provision.

Test plan

  • Agent deployed with FORGE_OTEL_ENDPOINT set emits spans visible in AIP's eval collector logs
  • Token refresh: long-running agent past the token's TTL still exports (refresh worked)
  • Endpoint unreachable → agent continues running, drop counter increments
  • Tenancy mismatch (span attributes vs token claims) → eval collector rejects, agent's exporter logs the rejection

Dependencies

  • initializ/eval#3 (collector), initializ/eval#4 (OIDC validation)

Metadata

Metadata

Assignees

No one assigned

    Labels

    epic:evalCross-repo observability + evaluation epic

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions