add client-side circuit breaker interceptor for gRPC clients

[k4leung4]

Summary

Add a new pkg/interceptors/circuitbreaker package that provides gRPC
client interceptors backed by sony/gobreaker. When a downstream service
returns too many consecutive errors, the circuit opens and calls fail fast
with codes.Unavailable instead of adding load to the failing service.

Motivation

Addresses CUS-239 /
CUS-172 /
chainguard-dev/internal-dev#18988.

During INC-44, all services continued retrying the overwhelmed load balancer,
amplifying the cascade. A circuit breaker would have stopped the cascade
within seconds by failing fast once the downstream was detected as unhealthy.

Package API

// Get default settings tuned for Cloud Run service-to-service calls.
settings := circuitbreaker.DefaultSettings("iam-datastore")
cb := gobreaker.NewCircuitBreaker[any](settings)

// Add to gRPC dial options (opt-in per client).
opts = append(opts,
    grpc.WithUnaryInterceptor(circuitbreaker.UnaryClientInterceptor(cb)),
    grpc.WithStreamInterceptor(circuitbreaker.StreamClientInterceptor(cb)),
)

Default settings

Parameter	Value	Rationale
ConsecutiveFailures to trip	5	Tolerates brief transient errors
Half-open timeout	15s	Gives downstream time to recover
Max probe requests (half-open)	10	Tests recovery with limited traffic
Failure count reset interval	30s	Prevents slow leak from tripping

Error classification

Client-side errors (the server didn't fail) are classified as successes
and do not count toward tripping the breaker:

• InvalidArgument, NotFound, AlreadyExists, PermissionDenied,
  Unauthenticated, FailedPrecondition, OutOfRange, Unimplemented, Canceled

Canceled is classified as a success because cancellation is typically
client-initiated (context timeout or user abort), not a server failure.

Server-side errors that count as failures:

• Internal, Unavailable, DeadlineExceeded, ResourceExhausted,
  Unknown, Aborted, DataLoss

Observability

State transitions (closed → open → half-open → closed) are logged via
clog.InfoContextf with context.Background() (circuit-level events,
not request-scoped).

Stream interceptor limitation

The stream interceptor only tracks stream establishment errors.
Errors on Send/Recv after the stream is established are not tracked,
so a downstream that accepts connections but fails on every message will
not trip the breaker. This is an inherent limitation of the gRPC stream
interceptor model.

Tests

• TestCircuitBreaker_TripsAfterConsecutiveFailures — circuit opens after
  N server errors, subsequent calls fail fast with Unavailable
• TestCircuitBreaker_ClientErrorsDoNotTrip — NotFound etc don't trip
• TestCircuitBreaker_RecoversThroughHalfOpen — open → half-open → probe
  succeeds → closed
• TestStreamClientInterceptor_TripsAndFailsFast — stream call fails
  fast when circuit is open
• TestDefaultSettings_IsSuccessful — table-driven test of error
  classification across 16 gRPC codes

Follow-up in mono

After bumping go-grpc-kit, enable the circuit breaker per-client in
api-impl/cmd/backend/main.go's configureClients(). Start with the
IAM datastore client (highest traffic, most critical) and expand.

Test plan

• go test ./pkg/interceptors/circuitbreaker/ -v — 5/5 pass
• CI passes
• Deploy to staging, verify state transition logs in Cloud Logging

🤖 Generated with Claude Code

[eslerm]

🤖: Solid circuit breaker implementation — no correctness bugs, good test coverage with real in-process server. Two comment nits.

[eslerm]

🤖: Comment says "context timeout" but codes.Canceled is explicit context cancellation (e.g., ctx.Cancel()), not a timeout. Timeouts produce codes.DeadlineExceeded which is already handled separately above. Consider: "cancelled by the client (explicit cancellation), not a server failure."

[eslerm]

🤖: Comment says "Trip after 2" but the condition is > 2, which trips after 3 consecutive failures. Should be "Trip after 3" or change to > 1.