Connect to storage backends with auth credentials (data proxy as OIDC provider)

[alukach]

Description

Implement outbound authentication for the data proxy to connect to upstream storage backends, as described in ADR-006.

The data proxy operates as an OIDC identity provider, issuing short-lived JWTs that upstream cloud providers (AWS, GCS, Azure) accept via workload identity federation. This eliminates the need to store long-lived cloud credentials in the proxy.

Key Requirements

• OIDC Issuer endpoints on data.source.coop:
  • /.well-known/openid-configuration — OIDC discovery document
  • JWKS endpoint — public keys for verifying tokens issued by the proxy
• Token issuance: Generate short-lived, audience-scoped JWTs for upstream providers
• Credential exchange: Exchange issued JWTs for cloud-native credentials at each provider's STS (AWS STS, GCP STS, Azure token endpoint)
• object_store crate adoption: Replace manual per-backend adapters with the unified ObjectStore trait
• Fallback: stored credentials — For backends that don't support OIDC federation, continue fetching static credentials from the Source Cooperative API (cached with short TTL)
• Data provider hosting: Support data providers registering their own upstream storage (S3 buckets, GCS buckets, etc.) with OIDC trust or stored credentials

References

• ADR-006: Outbound Connectivity — OIDC issuer model, object_store adoption, data provider hosting