feat(outbound): unified circuit breaker with success-rate and Retry-After awareness

[unleashed]

This PR is opened against #4537 to minimize the diff, and builds on top of it. It contains the
circuit breaker implementation before any changes to linkerd2-proxy-api affecting the API,
its messages, and the design of the circuit breaker. That is expected to happen after all the
original feature set is posted completely, as a set of changes on top.

N.B. there are still further upcoming changes building on top of this (ie. actually use the load biaser, last-N endpoint ejection safety mechanism).

Summary

This branch reworks outbound HTTP failure handling. It replaces the consecutive-failures
circuit breaker with a unified breaker that trips on either consecutive 5xx failures or a
declining EWMA success rate, and it teaches that breaker to honor server backpressure
signals (Retry-After, grpc-retry-pushback-ms). It also lands the building blocks for
failure-aware load balancing (a standalone linkerd-ewma crate and a
linkerd-load-biaser crate) and extends the client-policy configuration to carry the new
knobs from linkerd2-proxy-api 0.19.0.

Every new behavior is opt-in. With no policy configured, an endpoint gets a gate that never
shuts, no breaker task, no hint stores, and no load biasing: the prior behavior, unchanged.

What is live vs. staged

Capability	Status
Unified circuit breaker (consecutive + success-rate EWMA)	Active: wired per endpoint in the balancer's endpoint stack
Retry-After / grpc-retry-pushback-ms to breaker backoff floor	Active
linkerd-ewma crate	Active: powers the breaker's success-rate policy
Response-classification channel changes	Active
FailureAccrual (struct), SuccessRateConfig, RetryAfterConfig	Active: consumed by the breaker
linkerd-load-biaser crate	Staged: built and unit-tested, depended on, but not yet invoked by the balancer
LoadBiasConfig (load_bias)	Staged: parsed from proto and carried in policy types, not yet read by any stack layer

The balancer itself is unchanged on this branch (linkerd/proxy/balance has no diff); it
still uses Tower's PeakEwma. The load-biaser is a self-contained replacement prepared for
a follow-up that swaps the balancer's load metric. This is called out so the dormant
load_bias plumbing is not mistaken for an active code path.

Architecture

Component map

flowchart TD
    API["linkerd2-proxy-api 0.19.0 (proto)<br/>FailureAccrual, SuccessRate,<br/>LoadBiasConfig, RetryAfterConfig"]
    subgraph CP["linkerd-proxy-client-policy"]
        CPT["Http1 / Http2 / Grpc<br/>failure_accrual  (active)<br/>retry_after  (active)<br/>load_bias  (staged)"]
    end
    subgraph OUT["linkerd-app-outbound : http/breaker"]
        UNI["unified.rs: UnifiedBreaker state machine"]
        WRAP["wrap_classify.rs: NewRetryAfterGateSet<br/>per-endpoint gate, classifier, task"]
        RA["retry_after.rs: DurationHintStore,<br/>RetryAfterClassify"]
        BAL["concrete/balance.rs: wires gate-set<br/>beneath the PeakEwma balancer"]
    end
    subgraph LEAF["leaf crates"]
        EWMA["linkerd-ewma  (active)<br/>time-decayed average, get_at(now)"]
        CLS["linkerd-http-classify<br/>retry_after parsers,<br/>BroadcastClassification"]
        LB["linkerd-load-biaser  (staged)<br/>Load = max(rtt x (pending+1), penalty)"]
    end
    API -->|TryFrom + validation| CPT
    CPT -->|sidecar.rs / ingress.rs| OUT
    UNI -->|success rate| EWMA
    RA --> CLS
    WRAP --> RA
    BAL --> WRAP
    LB -. uses .-> EWMA
    LB -. not yet wired .-> BAL

Loading

Where the breaker sits in the outbound HTTP stack

The outbound stack resolves a logical service to a set of backends, and each backend to a
set of endpoint addresses behind a load balancer. The breaker lives at the bottom of that
balancer, one instance per resolved endpoint.

flowchart TD
    LOG["logical (per service): routes to backends"]
    CONC["concrete (per backend)"]
    BALANCER["NewBalance: P2C over PeakEwma<br/>selects a ready endpoint"]
    LOG --> CONC --> BALANCER
    BALANCER -->|one gated stack per endpoint| E1
    BALANCER --> E2
    BALANCER --> E3
    subgraph E1["endpoint 10.0.0.1"]
        direction TB
        G1["svc::Gate (admission)"] --> C1["Insert + BroadcastClassification"] --> H1["HTTP client"]
        T1[["UnifiedBreaker task"]]
    end
    subgraph E2["endpoint 10.0.0.2"]
        direction TB
        G2["svc::Gate"] --> C2["Insert + BroadcastClassification"] --> H2["HTTP client"]
        T2[["UnifiedBreaker task"]]
    end
    subgraph E3["endpoint 10.0.0.3"]
        direction TB
        G3["svc::Gate"] --> C3["Insert + BroadcastClassification"] --> H3["HTTP client"]
        T3[["UnifiedBreaker task"]]
    end
    C1 -. classify::Class .-> T1
    T1 -. gate state .-> G1
    C2 -. classify::Class .-> T2
    T2 -. gate state .-> G2
    C3 -. classify::Class .-> T3
    T3 -. gate state .-> G3

Loading

concrete/balance.rs pushes breaker::NewRetryAfterGateSet onto the endpoint stack before
pushing http::NewBalance. In Tower, the later push is the outer layer, so the balancer
wraps a set of already-gated endpoints. Because NewBalance instantiates the inner
NewService once per resolved endpoint, each endpoint gets its own gate, its own
classification channel, its own hint stores, and its own breaker task. A trip on one
endpoint never affects another.

This reuses the proxy's existing endpoint-ejection model. The breaker does not reject
requests directly; it actuates a svc::Gate (linkerd/stack/src/gate.rs), and the gate's
readiness is what the balancer observes:

• Open: poll_ready succeeds, so the endpoint is selectable by P2C.
• Shut: poll_ready stays Pending, so the balancer's readiness cache drops the
  endpoint from the selectable set and traffic shifts to healthy peers.
• Limited(Semaphore(1)): exactly one request is admitted; this is the recovery probe.

Construction-time wiring (per endpoint)

flowchart LR
    A["FailureAccrual + RetryAfterConfig"] --> B["NewRetryAfterGateSet::new_service<br/>(per endpoint)"]
    B --> C["build RetryAfterStore +<br/>GrpcRetryPushbackStore"]
    B --> D["create gate channel +<br/>classification mpsc"]
    B --> E["spawn UnifiedBreaker::run()"]
    B --> F["assemble: Gate to InsertRetryAfterClassify<br/>to BroadcastClassification to HTTP client"]

Loading

When the resolved accrual is absent, or present but unable to ever trip, the endpoint
takes a no-breaker path instead: the stock BroadcastClassification and a gate that never
shuts, with no stores and no task. The "unable to ever trip" filter
(is_effectively_disabled) catches a policy with max_failures == 0 and no usable
success-rate threshold, pinning its cost to that of having no circuit breaking.

Configuration flow

Configuration is validated once at the proto boundary and then carried as plain target
params down to the point where the breaker task is spawned.

flowchart TD
    P["linkerd2-proxy-api 0.19.0<br/>outbound.proxy_protocol Http1 / Http2 / Grpc"]
    CP["client-policy types<br/>failure_accrual: Option&lt;FailureAccrual&gt;<br/>retry_after: Option&lt;RetryAfterConfig&gt;<br/>load_bias: Option&lt;LoadBiasConfig&gt; (staged)"]
    R["logical policy router<br/>builds Concrete&lt;T&gt; with failure_accrual + retry_after"]
    B["Balance&lt;T&gt; param impls<br/>HasFailureAccrual<br/>Param Option&lt;RetryAfterConfig&gt;<br/>Param EwmaConfig"]
    G["NewRetryAfterGateSet::new_service<br/>reads accrual + max_duration<br/>filters effectively-disabled policies"]
    U["UnifiedBreakerConfig<br/>spawn UnifiedBreaker::run()"]
    P -->|TryFrom + validation| CP
    CP -->|sidecar.rs / ingress.rs extract per protocol| R
    R --> B --> G --> U

Loading

Validation happens in the TryFrom impls in client-policy: the success-rate threshold
must be within [0.0, 1.0] (which also rejects NaN), decay must be at least the EWMA
floor (MIN_DECAY, 1 ms) and defaults to 10 s when absent, and min_requests is bounded so
cold-start can always be satisfied. Those range checks reject out-of-bounds values at
conversion with an InvalidValue error. A config that is in range yet can never trip
(threshold == 0.0, say) is not rejected at conversion; it is accepted and then collapsed to
the no-breaker path at stack-build time by is_effectively_disabled (wrap_classify.rs),
rather than producing a live breaker that can never trip.

Concrete<T> carries failure_accrual and retry_after, but its Eq/Hash are
hand-written to exclude them. Two reasons: SuccessRateConfig holds an f64, which has no
total Hash/Eq; and breaker configuration is not part of backend identity: it controls
how a backend is treated, not which backend it is. The backend cache key stays
{target, authority, parent, parent_ref, backend_ref}.

The unified circuit breaker

The breaker is a single async task per endpoint (UnifiedBreaker::run) that owns a
three-state machine. It consumes classify::Class verdicts from an mpsc channel and drives
the gate.

stateDiagram-v2
    [*] --> Open: start (EWMA = 1.0, counters = 0)
    Open --> Closed: TRIP
    Closed --> Probation: backoff elapsed
    Probation --> Open: probe succeeds (reset state)
    Probation --> Closed: probe fails (advance backoff)
    note right of Open
        gate.open() - all requests admitted.
        Per response: update consecutive (5xx only),
        EWMA (5xx or 429 or RESOURCE_EXHAUSTED scores 0.0),
        request_count; reset count after idle beyond 3x decay.
        TRIP when consecutive reaches max_failures
        OR (count reaches min_requests AND EWMA below threshold).
    end note
    note right of Closed
        gate.shut() - endpoint ejected from the balancer.
        Wait max(backoff_step, hint); the first wait is
        floored by max(Retry-After, gRPC pushback), capped.
    end note
    note left of Probation
        gate.limit(1) - exactly one probe admitted.
        Dual mode: probe must be non-5xx AND non-429.
        Consecutive-only mode: class.is_success().
    end note

Loading

Two policies, one circuit. Consecutive-failure tracking counts configured failures
(typically 5xx) and resets on any success; it has no cold-start protection because repeated
hard failures are a strong signal at any sample size. The success-rate policy feeds an EWMA
where both 5xx and 429 / gRPC RESOURCE_EXHAUSTED count as failure, so rate limiting drags
the rate down. Either policy can trip the circuit. When a single response crosses both
thresholds at once, the trip is attributed to consecutive failures because that check runs
first; the TripReason is for observability only.

Cold-start protection applies to the success-rate policy in three layers: the EWMA
initializes optimistically at 1.0; the circuit cannot trip on success rate until
min_requests responses have been seen; and after an idle gap longer than 3 x decay the
request counter resets, because at that point a single new sample would dominate the decayed
average and could trip the circuit on its own. Idle is measured against the last response
time, tracked separately from the EWMA whose timestamp freezes while the circuit is shut and
probing.

Recovery is mode-aware, which is how backwards compatibility is preserved:

success_rate	max_failures	Mode	Probe success test
None	> 0	consecutive-only	class.is_success(), exactly the prior breaker's semantics
Some	0	success-rate-only	non-5xx and non-429 (HTTP) / Code::Ok (gRPC)
Some	> 0	dual / unified	as success-rate-only (stricter)
None / threshold <= 0	0	inert, treated as no breaker	n/a
field absent	n/a	no breaker	gate never shuts, no task spawned

The mode is fixed at construction, not re-derived at recovery: when the success-rate policy
is absent, wrap_classify.rs sets min_requests = usize::MAX (and threshold = 0.0), and
the breaker keys probe behavior off that min_requests == usize::MAX sentinel rather than
re-checking success_rate.is_some().

In consecutive-only mode the breaker delegates the probe to the default classifier, so a 429
is judged exactly as the old breaker judged it. In any mode with success-rate active, a 429
during probation is treated as failure, because reopening to a still-rate-limited endpoint
would immediately re-trip.

Request and response data flow

Two channels connect the request path to the breaker task, both created per endpoint at
construction time: a gate channel (watch<State>) the breaker writes, and an mpsc channel
of classifications the breaker reads.

sequenceDiagram
    autonumber
    participant BAL as Balancer (P2C)
    participant G as svc::Gate
    participant I as InsertRetryAfterClassify
    participant B as BroadcastClassification
    participant U as Upstream endpoint
    participant S as Hint stores
    participant CB as UnifiedBreaker task
    Note over G,CB: gate channel and classification mpsc are created per endpoint
    BAL->>G: poll_ready (pick a ready endpoint)
    G->>G: admit only if Open or holding a Limited permit
    G->>I: request
    I->>I: read classify::Response from extensions,<br/>wrap as RetryAfterClassify, re-insert
    I->>B: request
    B->>U: request (response body wrapped)
    U-->>B: response / trailers / error
    B->>S: drives RetryAfterClassify / GrpcRetryPushbackClassifyEos,<br/>which record Retry-After (429/503) and grpc-retry-pushback-ms<br/>(RESOURCE_EXHAUSTED, gRPC headers only on HTTP 200 OK)
    B->>CB: try_send(classify::Class)
    B-->>I: response
    I-->>G: response
    G-->>BAL: response
    CB->>CB: update consecutive, EWMA, counters
    alt policy breached
        CB->>S: drain hints for the backoff floor
        CB->>G: gate.shut() (endpoint ejected)
    end

Loading

The hint stores (DurationHintStore, an Arc<Mutex<Option<(Instant, Duration)>>>) decouple
the classifier from the breaker. Recording is max-value-wins; take(max_age) returns the
remaining wait (recorded duration minus elapsed) only if the hint is still fresh, consumes
it so it is used once, and discards stale entries. Backpressure on the classification channel
is non-blocking: try_send failures are logged at debug and dropped rather than stalling the
response path. The breaker also drains the channel while shut, so a closed channel ends the
task instead of spinning.

Retry-After parsing handles both the delay-seconds and HTTP-date forms (429/503 only); gRPC
pushback parsing reads grpc-retry-pushback-ms, treating a negative value as "do not retry"
(no hint). Unary gRPC failures carry the pushback in headers (parsed at start); streaming
failures carry it in trailers (parsed at eos via a GrpcRetryPushbackClassifyEos wrapper).
All hints are capped by the per-endpoint RetryAfterConfig.max_duration, defaulting to
DEFAULT_RETRY_AFTER_MAX_DURATION (300 s).

Supporting crates

linkerd-ewma is a standalone time-decayed exponentially-weighted moving average. The
reason it exists separately from Tower's internal RTT estimator is get_at(now): a
non-mutating, time-projected read (value * exp(-elapsed/decay)), which lets the breaker
sample the success rate without perturbing it, and lets the future load-biaser read load
under a shared lock. add blends with a time-aware alpha and drops samples that share a
timestamp with the last update. The breaker initializes it with
new_with_value(decay, now, 1.0).

linkerd-load-biaser (staged) wraps a service to track an RTT EWMA and a penalty EWMA,
exposing Load = max(rtt * (pending + 1), penalty) for P2C selection. It classifies
responses into FailureHint::{RateLimited, ServiceUnavailable, InternalError} across HTTP
and gRPC, and amplifies the penalty when a Retry-After / pushback hint is present so a
rate-limited endpoint stays de-prioritized through the server's stated window. It is
unit-tested in isolation; wiring it into the balancer in place of Tower PeakEwma is left to
a follow-up, which is also where LoadBiasConfig becomes live.

Backwards Compatibility

• All new client-policy fields are optional; absent proto fields convert to None
• Feature requires explicit policy configuration to activate
• Default behavior unchanged without opt-in (gate never shuts, no breaker task, no load biasing, hints ignored)
• FailureAccrual enum-to-struct migration preserves the no-accrual semantics (absent accrual still yields a gate that never shuts)
• Consecutive-only mode reproduces the prior breaker's trip and probe semantics
• linkerd/proxy/balance is unchanged; load-balancing behavior is unchanged
• Endpoint-derived (non-policy) backends set the new fields to None explicitly (discover.rs)

Proto wire compatibility (fresh field numbers, optional fields) is owned by the companion
linkerd2-proxy-api branch and is verified there, not in this repo.

Testing

• Breaker unit tests in unified.rs cover starts-open, consecutive trips, success resetting
  the consecutive count, cold-start and idle re-arming, retry-after backoff flooring, and the
  dual / consecutive-only probe semantics.
• breaker/integration_tests.rs drives the breaker task against the real svc::Gate,
  feeding classifications through the per-endpoint channel: trip-and-recover, HTTP
  Retry-After and gRPC pushback extending backoff, and a combined max-value-wins case. Two
  cases (*_end_to_end) route an actual response through the real classifier into the hint
  store; the rest pre-seed the store directly. (Cold-start re-arming after idle is covered in
  unified.rs, above, not here.)
• Balancer-level tests in http/logical/tests/failure_accrual.rs exercise the full outbound
  stack: consecutive-failure accrual, the balancer dropping tripped endpoints from selection
  (balancer_doesnt_select_tripped_breakers), and per-endpoint Retry-After isolation with
  no cross-endpoint bleed.
• Store-level tests in breaker/retry_after.rs check max-value-wins, staleness, and
  elapsed-adjustment behavior.
• Client-policy tests cover the proto conversions, including rejection of out-of-range
  success-rate configs (threshold, decay, and min_requests bounds).
• The leaf crates (ewma, load-biaser, classify retry_after) are independently
  unit-tested.

Reviewing by commit groups

The history is ordered to build bottom-up and is reviewable in roughly these groups:

1. Leaf crates: ewma, load-biaser, classify retry_after parsers, classify channel API.
2. Config: client-policy FailureAccrual enum-to-struct migration and new config types,
   proto bump to 0.19.0.
3. Breaker: duration hint stores, the unified state machine, the classify gate set, and the
   NewRetryAfterGateSet wiring.
4. Stack integration: concrete/balance.rs, logical*, sidecar.rs, ingress.rs.
5. Tests: unit, regression, and integration coverage.

Cross-Repo Dependencies

Requires linkerd2-proxy-api 0.19.0, which introduces the SuccessRate, LoadBiasConfig,
and RetryAfterConfig proto messages and the FailureAccrual struct fields used by the
config conversions.