Outlier detection behavior for client cancellations and hedged streams

[shivamgupta1]

The stream tracer for outlier detection seems to record each stream closure with status.isOk() != true as an error. The rate of success is then used for success rate and failure percentage outlier detectors.

I'd like to understand more as to what happens in the following cases:

• A hedged stream is cancelled because another winning stream was committed.
• The client cancels the call due to context expiration or any other reason unrelated to the call itself.

Would it still be recorded as an error and accounted for outlier detection?

[AgraVator]

Hello @shivamgupta1,

Current Implementation

The OutlierDetectionLoadBalancer tracks call results using a ClientStreamTracer. Specifically, it intercepts the streamClosed event and checks if the status is OK.

Here is the relevant code snippet from OutlierDetectionLoadBalancer.java:

            @Override
            public void streamClosed(Status status) {
              tracker.incrementCallCount(status.isOk());
              delegate().streamClosed(status);
            }

The tracker.incrementCallCount method (found in OutlierDetectionLoadBalancer.java) then increments either the success or failure count based on that boolean:

    void incrementCallCount(boolean success) {
      // ...
      if (success) {
        activeCallCounter.successCount.getAndIncrement();
      } else {
        activeCallCounter.failureCount.getAndIncrement();
      }
    }

Since status.isOk() only returns true for Status.OK, any non-OK status is recorded as an error and accounted for in outlier detection (both for Success Rate and Failure Percentage algorithms).

Now, for the cases u highlighted

1. A hedged stream is cancelled because another winning stream was committed:
   • When a hedged stream is cancelled, it typically terminates with a CANCELLED status. Since Status.CANCELLED.isOk() is false, it will be recorded as an error for outlier detection.
2. The client cancels the call due to context expiration or any other reason unrelated to the call itself:
   • If the client cancels the call (status CANCELLED) or if the context expires (status DEADLINE_EXCEEDED), these statuses are also non-OK. Therefore, they will also be recorded as errors for outlier detection.

[shivamgupta1]

Hi @AgraVator, thanks for checking. We use the same logic in our custom loadbalancer. On enabling hedging recently for some apps, we observed that outlier detection was always activated, i.e., some servers were being ejected erroneously all the time. Hedged stream and client cancellations (perhaps with the exception of deadline exceeded) shouldn't count towards server failures right?

[shivamgupta1]

Hi @AgraVator, do you mind checking this again? Thank you.