Route promotes a Revision to 100% traffic while it is still below initial-scale (latestReadyRevision latches an under-scaled Revision)

[juanpark-dandy]

/area autoscale
/area API

/kind spec

What version of Knative?

Knative Serving v1.20.0 (observed). Root-cause code is unchanged on main and in the latest
release knative-v1.22.1, so all released versions through v1.22.1 are affected.
Related (not duplicates): #11531, #2674, #11373.

Expected Behavior

Per the scale-bounds docs,
initial-scale is "the initial target scale a Revision must reach ... before it is marked as Ready."
A new Revision should not become Ready, should not be promoted to
Configuration.status.latestReadyRevisionName, and should not receive route traffic until it reaches
its initial-scale. Traffic should stay on the previous, fully-scaled Revision until then.

Actual Behavior

A new Revision can be marked Ready=True and latched as latestReadyRevisionName while still far below
its initial-scale, so the Route shifts 100% of traffic to an under-provisioned Revision. Because
latestReadyRevisionName is monotonic, the route never reverts — even after the Revision later flips
Ready=False (e.g. ProgressDeadlineExceeded). In production this abandoned a healthy fully-scaled
Revision for a new one running at ~15% of target replicas, which then returned 503/504s.

Root cause (same on release-1.20 and main):

• Revision readiness is not gated on initial-scale —
  pkg/apis/serving/v1/revision_lifecycle.go:
  revisionCondSet = apis.NewLivingConditionSet(ResourcesAvailable, ContainerHealthy) (no
  ScaleTargetInitialized/Active term).

• ResourcesAvailable comes from the Deployment's Progressing condition, not Available —
  pkg/reconciler/revision/reconcile_resources.go PropagateDeploymentStatus →
  TransformDeploymentStatus reads {Progressing, ReplicaSetReady}, never appsv1.DeploymentAvailable.
  So ResourcesAvailable=True at any replica count, and ContainerHealthy=True once ReadyReplicas > 0.

• The only step that holds Ready back below initial-scale (PropagateAutoscalerStatus in reconcilePA,
  which pulls ResourcesAvailable back to Unknown while !IsScaleTargetInitialized()) is skipped
  when an earlier phase errors. The reconcile is an early-return phase loop
  (pkg/reconciler/revision/revision.go):

  for _, phase := range []func(context.Context, *v1.Revision) error{
      c.reconcileDeployment,   // sets ResourcesAvailable=True (Progressing) + ContainerHealthy=True
      c.reconcileImageCache,   // if this errors, the loop returns...
      c.reconcilePA,           // ...so reconcilePA (which would reset ResourcesAvailable) is SKIPPED
  } {
      if err := phase(ctx, rev); err != nil { return err }
  }
  
  A transient reconcileImageCache error (e.g. createImageCache AlreadyExists from image-lister lag,
  common under heavy concurrent-write churn) persists the below-initial-scale Ready=True. Then
  pkg/reconciler/configuration/configuration.go findAndSetLatestReadyRevision advances
  latestReadyRevisionName to it and never reverts (monotonic); the Route follows.
  (Note: a reconcileDeployment Update conflict does NOT trigger this — it returns before
  PropagateDeploymentStatus, so ResourcesAvailable is never set True that cycle.)

Steps to Reproduce the Problem

In production this fires stochastically under churn. To make it deterministic, the steps below
(a) wedge a new Revision below initial-scale and (b) force reconcileImageCache to error via a
ResourceQuota (a stand-in for the transient AlreadyExists). Requires image caching enabled
(caching.internal.knative.dev Image CRD + controller).

1. Label 3 schedulable nodes and create a namespace:
   kubectl label node repro=true
   kubectl create ns repro
2. Deploy a healthy baseline Revision at scale 1, pinned to the labeled nodes; wait until Ready:
   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata: { name: rt, namespace: repro }
   spec:
   template:
   metadata:
   annotations:
   autoscaling.knative.dev/initial-scale: "1"
   autoscaling.knative.dev/min-scale: "1"
   autoscaling.knative.dev/max-scale: "1"
   spec:
   nodeSelector: { repro: "true" }
   containers:
   - image: ghcr.io/knative/helloworld-go:latest
   env: [{ name: TARGET, value: "v1" }]
3. Cap the namespace's Image count at its current value so the next Revision's image cache can't be created:
   kubectl -n repro create quota capimg
   --hard=count/images.caching.internal.knative.dev=$(kubectl -n repro get images.caching.internal.knative.dev --no-headers | wc -l)
4. Roll a new Revision that wants initial-scale=6 but can only schedule 2 pods (one per labeled node;
   one node is held by the baseline) — wedging it at 2/6:

re-apply the same Service with:

spec:
template:
metadata:
annotations:
autoscaling.knative.dev/initial-scale: "6"
autoscaling.knative.dev/min-scale: "6"
autoscaling.knative.dev/max-scale: "6"
spec:
nodeSelector: { repro: "true" }
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels: { serving.knative.dev/service: rt }
containers:
- image: ghcr.io/knative/helloworld-go:latest
env: [{ name: TARGET, value: "v2" }] # forces a new Revision
5. Observe the new Revision become Ready=True and capture the route while below initial-scale:
kubectl -n repro get revision # new rev READY=True
kubectl -n repro get pods # 2 Running, 4 Pending (wedged at 2/6)
kubectl -n repro get images.caching.internal.knative.dev # only the baseline's image exists
kubectl -n repro get configuration rt -o jsonpath='{.status.latestReadyRevisionName}' # -> new rev
kubectl -n repro get route rt -o jsonpath='{.status.traffic}' # -> 100% new rev

5. The new Revision is Ready=True at 2/6 replicas with no image cache; latestReadyRevisionName and the
   Route both point at it, and never revert to the fully-scaled baseline.

Suggested fix

Make readiness reflect replica sufficiency: include the Deployment's Available condition in
TransformDeploymentStatus, or gate Revision Ready / findAndSetLatestReadyRevision advancement on the
PodAutoscaler's ScaleTargetInitialized (enforcing the documented initial-scale semantics). Either
prevents latestReadyRevisionName from advancing to an under-scaled Revision.

[juanpark-dandy]

Script I used to reproduce the premature route switch.

It doesn't seem to happen all the time, so far I have been able to reproduce it ~4/5 attempts with the script.

#!/usr/bin/env bash
#
# reproduce-early-route-capture.sh  (v8 — mechanism B: below-initial-scale route capture)
#
# Reproduces the exact routing failure behind the 2026-06-16 teeth-segmentation outage: on an
# image bump, the KServe/Knative Route promotes 100% of traffic to the NEW revision while that
# revision is STILL BELOW its initial-scale (prod: ~3 ready of 20). The under-scaled, overwhelmed
# revision then 504s under the model's tight 60s timeout while the healthy prior revision — which
# still had all its replicas — is abandoned. This is the "route points to a revision that is not
# fully scaled up yet" bug, reproduced DETERMINISTICALLY.
#
# ── Mechanism, traced in knative/serving release-1.20 (your cluster's version) ──
#   The Revision reconciler runs a phase loop with early-return-on-error (revision.go):
#       [ reconcileDeployment, reconcileImageCache, reconcilePA ]
#     reconcileDeployment  -> PropagateDeploymentStatus sets ResourcesAvailable=True from the
#                             Deployment's *Progressing* status (NOT Available; reconcile_resources.go),
#                             and ReadyReplicas>0 sets ContainerHealthy=True.
#     reconcileImageCache  -> createImageCache(); if it returns an error the loop RETURNS, so the next
#                             phase (reconcilePA) is SKIPPED.
#     reconcilePA          -> PropagateAutoscalerStatus, which is the ONLY thing that pulls
#                             ResourcesAvailable back to Unknown while below initial-scale
#                             (!ScaleTargetInitialized). When it's skipped, the partial-scale
#                             Ready=True (ResourcesAvailable ∧ ContainerHealthy) PERSISTS.
#   Revision Ready = NewLivingConditionSet(ResourcesAvailable, ContainerHealthy) (revision_lifecycle.go)
#   -- it is NOT gated on initial-scale. So a skipped reconcilePA => Revision Ready=True at, say, 2/6 =>
#   Configuration.latestReadyRevisionName latches it (configuration.go, MONOTONIC, never reverts) =>
#   the Route (traffic latestRevision:true) follows it to the under-scaled revision and never moves back.
#
# ── Why this is a RACE in prod but DETERMINISTIC here ──
#   In prod the reconcileImageCache error was a transient createImageCache "AlreadyExists" (image-lister
#   lag) that only happened because 33 simultaneous rollouts + GPU-stockout scheduling churn hammered
#   the controllers/apiserver with concurrent writes. A quiet cluster never produces that conflict.
#   We induce the SAME createImageCache failure deterministically and gently — a ResourceQuota that caps
#   the Image count in the throwaway namespace, so the new revision's Image simply cannot be created.
#   Same code path, same latch; no load on shared infra. (To instead watch the natural race, you'd need
#   prod-scale concurrent-write pressure — not reproducible from a quiet cluster.)
#
#   Under-scale is made deterministic too: pin both revisions to N_NODES labeled nodes with one-pod-per-
#   node anti-affinity, so the new revision wedges at exactly (N_NODES-1) ready, below initial-scale.
#
# GPU-free. SAFE: throwaway namespace + temporary node labels, both removed on exit.
#
# Exit codes:  0 = PASS   1 = NO-REPRO   2 = INCONCLUSIVE (env)
#
# Usage:  CONTEXT=<ctx> ./reproduce-early-route-capture.sh
#   env: NS IMAGE INITIAL_SCALE N_NODES CPU POLL_TIMEOUT EXPECT KEEP
#        INITIAL_SCALE=6  -> min==max==initial-scale of the rev (mirror prod teeth_seg = 20)
#        N_NODES=3        -> labeled nodes; new rev wedges at (N_NODES-1) ready (must be >=3 -> >=2 ready)
#        EXPECT=captured (default) -> PASS when the route is promoted below initial-scale (pre-fix behavior)
#        EXPECT=held               -> PASS when the route holds on the prior rev (post-fix guard:
#                                     e.g. gate latestReadyRevision on ScaleTargetInitialized, or canaryTrafficPercent)
#        KEEP=oncapture|1          -> keep ns + labels for inspection IF reproduced; KEEP=always -> never tear down
set -uo pipefail   # NOT -e: a transient control-plane blip must not abort a long poll

CONTEXT="${CONTEXT:-$(kubectl config current-context)}"
NS="${NS:-route-capture-repro-$RANDOM}"
IMAGE="${IMAGE:-python:3.11-alpine}"
INITIAL_SCALE="${INITIAL_SCALE:-6}"        # min == max == initial-scale (mirror prod teeth_seg: set to 20)
N_NODES="${N_NODES:-3}"                     # new rev wedges at (N_NODES-1) ready; need >=3 so ready>=2
CPU="${CPU:-0.1}"
POLL_TIMEOUT="${POLL_TIMEOUT:-180}"
EXPECT="${EXPECT:-captured}"
KEEP="${KEEP:-}"
LABEL_KEY="route-capture-repro"
k() { kubectl --context "$CONTEXT" "$@"; }
# resilient read-only query: retries transient control-plane blips, never aborts the script
kq() { local o; for _ in 1 2 3 4; do if o="$(kubectl --context "$CONTEXT" "$@" 2>/dev/null)"; then printf '%s' "$o"; return 0; fi; sleep 2; done; printf ''; return 0; }

NODES=(); _keep=0; [ "$KEEP" = "always" ] && _keep=1
cleanup() {
  if [ "$_keep" = "1" ]; then
    echo "── KEEPING resources (KEEP=$KEEP) ──"
    echo "  inspect : kubectl --context $CONTEXT -n $NS get isvc,route,revision,resourcequota,pods -o wide"
    echo "  teardown: kubectl --context $CONTEXT delete ns $NS"
    [ -n "${NODES[*]:-}" ] && echo "            for n in ${NODES[*]}; do kubectl --context $CONTEXT label node \$n ${LABEL_KEY}-; done"
    return
  fi
  echo "── cleanup ──"
  k delete ns "$NS" --wait=false >/dev/null 2>&1 || true
  for n in "${NODES[@]:-}"; do k label node "$n" "${LABEL_KEY}-" >/dev/null 2>&1 || true; done
}
trap cleanup EXIT

echo "context: $CONTEXT   expect: $EXPECT   initial-scale: $INITIAL_SCALE   nodes: $N_NODES (-> new rev wedged at $((N_NODES-1))/$INITIAL_SCALE)"
[ "$N_NODES" -ge 3 ] || { echo "ERROR: N_NODES must be >=3 (need >=2 ready on the new rev)"; exit 2; }
[ "$INITIAL_SCALE" -gt "$((N_NODES-1))" ] || { echo "ERROR: INITIAL_SCALE must exceed N_NODES-1 (else not under-scaled)"; exit 2; }
k get ns knative-serving >/dev/null 2>&1 || { echo "ERROR: knative-serving not found"; exit 2; }
k get ns kserve >/dev/null 2>&1 || k get ns kserve-system >/dev/null 2>&1 || { echo "ERROR: kserve not found"; exit 2; }
k get crd images.caching.internal.knative.dev >/dev/null 2>&1 || { echo "ERROR: image caching CRD absent; the deterministic trigger needs it"; exit 2; }

echo "── selecting $N_NODES non-GPU schedulable nodes ──"
cat > /tmp/.route-repro-pick.py <<PY
import json,subprocess,sys
ctx="$CONTEXT"; want=int("$N_NODES")
nodes=json.loads(subprocess.check_output(["kubectl","--context",ctx,"get","nodes","-o","json"]))["items"]
out=[]
for n in nodes:
    a=n["status"].get("allocatable",{})
    if a.get("nvidia.com/gpu") not in (None,"0",0): continue
    if any(t.get("effect")=="NoSchedule" for t in n["spec"].get("taints",[])): continue
    if n["spec"].get("unschedulable"): continue
    out.append(n["metadata"]["name"])
for nm in out[:want]: print(nm)
PY
mapfile -t NODES < <(python3 /tmp/.route-repro-pick.py)
[ "${#NODES[@]}" -ge "$N_NODES" ] || { echo "ERROR: found ${#NODES[@]} schedulable non-GPU nodes, need $N_NODES."; exit 2; }
for n in "${NODES[@]}"; do echo "  pin $n"; k label node "$n" "${LABEL_KEY}=true" --overwrite >/dev/null; done

k create ns "$NS" >/dev/null 2>&1 || true
k label ns "$NS" istio-injection=enabled --overwrite >/dev/null 2>&1 || true

# trivial fast server: /ready passes ~2s after start; / returns 200 identifying the revision.
read -r -d '' SERVER_PY <<'PY' || true
import time, http.server, socketserver, os
START = time.time(); TARGET = os.environ.get("TARGET", "?")
class H(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        if self.path.startswith("/ready"):
            ok = (time.time() - START) >= 2
            self.send_response(200 if ok else 503); self.end_headers(); self.wfile.write(b"ok" if ok else b"x"); return
        self.send_response(200); self.end_headers(); self.wfile.write(("served-by=" + TARGET + "\n").encode())
class S(socketserver.ThreadingMixIn, http.server.HTTPServer): daemon_threads = True
S(("0.0.0.0", 8080), H).serve_forever()
PY
SERVER_B64="$(printf '%s' "$SERVER_PY" | base64 | tr -d '\n')"

# $1=TARGET  $2=scale(min==max==initial-scale)  $3=antiAffinity(0|1)
isvc_yaml() {
  local aff="" ann=""
  [ "$3" = 1 ] && aff="
    affinity: {podAntiAffinity: {requiredDuringSchedulingIgnoredDuringExecution: [{topologyKey: kubernetes.io/hostname, labelSelector: {matchLabels: {serving.knative.dev/configuration: rt-predictor}}}]}}"
  [ "$2" -gt 1 ] && ann="
    autoscaling.knative.dev/initial-scale: \"$2\""   # chart sets initial-scale == minReplicas when min > 1
  cat <<EOF
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: rt
  namespace: ${NS}
  labels: {sidecar.istio.io/inject: "true"}
  annotations: {${ann}}
spec:
  predictor:
    minReplicas: $2
    maxReplicas: $2
    containerConcurrency: 1
    nodeSelector: {${LABEL_KEY}: "true"}
    containers:
      - name: kserve-container
        image: ${IMAGE}
        ports: [{containerPort: 8080, protocol: TCP}]
        command: ["/bin/sh","-c"]
        args: ["echo ${SERVER_B64} | base64 -d > /tmp/s.py && exec python3 /tmp/s.py"]
        env: [{name: TARGET, value: "$1"}]
        readinessProbe: {httpGet: {path: /ready, port: 8080}, initialDelaySeconds: 1, periodSeconds: 2, failureThreshold: 90}
        resources: {requests: {cpu: "${CPU}"}, limits: {cpu: "${CPU}"}}$aff
EOF
}

curl_route() {
  k -n "$NS" run "c$RANDOM" --image=curlimages/curl --restart=Never --attach --rm -q --command -- \
    sh -c "curl -s -m 8 -w ' %{http_code}' http://rt.${NS}.svc.cluster.local/ 2>/dev/null | tr -d '\n' || echo ' 000'" 2>/dev/null
}

# rev1: healthy baseline (scale 1) — takes one node, creates its Image, serves 200.
echo "── rev1 baseline ──"
isvc_yaml rev1 1 0 | k apply -f - >/dev/null
if ! k -n "$NS" wait --for=condition=Ready isvc/rt --timeout=240s >/dev/null 2>&1; then
  echo "INCONCLUSIVE: rev1 baseline didn't become Ready"; exit 2
fi
echo "  rev1 ready; route serves [$(curl_route) ]"

# Cap the Image count at its current value so the NEW revision's createImageCache fails deterministically
# -> reconcileImageCache errors every reconcile -> reconcilePA is skipped -> the below-scale Ready latches.
imgcount="$(kq -n "$NS" get images.caching.internal.knative.dev --no-headers | grep -c .)"; imgcount="${imgcount:-1}"
echo "── capping count/images.caching.internal.knative.dev at $imgcount (forces new-rev image-cache failure) ──"
cat <<EOF | k apply -f - >/dev/null
apiVersion: v1
kind: ResourceQuota
metadata: {name: capimg, namespace: ${NS}}
spec:
  hard: {count/images.caching.internal.knative.dev: "${imgcount}"}
EOF

# rev2: the rollout — wants INITIAL_SCALE, anti-affinity wedges it at (N_NODES-1) ready, and its Image
# can't be created (quota) so reconcileImageCache keeps erroring -> reconcilePA skipped -> latch.
echo "── rev2 rollout (wedged below initial-scale; image-cache create blocked) ──"
isvc_yaml rev2 "$INITIAL_SCALE" 1 | k apply -f - >/dev/null
NEWREV=""; d=$(( $(date +%s) + 60 ))
while [ "$(date +%s)" -lt "$d" ]; do
  NEWREV="$(kq -n "$NS" get isvc rt -o jsonpath='{.status.components.predictor.latestCreatedRevision}')"
  [ -n "$NEWREV" ] && [ "$NEWREV" != "rt-predictor-00001" ] && break; sleep 2
done
echo "  baseline=rt-predictor-00001  new=$NEWREV"

# count the new rev's actually-Ready pods directly (revision .status.actualReplicas is populated by
# PropagateAutoscalerStatus — the very step being skipped — so it stays empty; the pods are the truth).
ready_pods() { kq -n "$NS" get pods -l serving.knative.dev/revision="$NEWREV" \
  -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c True; }

d=$(( $(date +%s) + POLL_TIMEOUT )); lr=""; rp=0; maxrp=0
while [ "$(date +%s)" -lt "$d" ]; do
  lr="$(kq -n "$NS" get isvc rt -o jsonpath='{.status.components.predictor.latestReadyRevision}')"
  rp="$(ready_pods)"; rp="${rp:-0}"
  [ "$rp" -gt "$maxrp" ] 2>/dev/null && maxrp="$rp"
  printf '\r  latestReady=%s  new ready-pods=%s/%s (peak %s)      ' "$lr" "$rp" "$INITIAL_SCALE" "$maxrp"
  [ "$lr" = "$NEWREV" ] && break
  [ "$rp" -ge "$INITIAL_SCALE" ] 2>/dev/null && break
  sleep 2
done
echo

newready="$(kq -n "$NS" get revision "$NEWREV" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')"
route="$(kq -n "$NS" get route rt-predictor -o jsonpath='{range .status.traffic[*]}{.revisionName}={.percent}% {end}')"
imgs="$(kq -n "$NS" get images.caching.internal.knative.dev -l serving.knative.dev/revision="$NEWREV" --no-headers | grep -c .)"
served="$(curl_route)"
echo "  new ready-pods=$maxrp/$INITIAL_SCALE  newRevReady=$newready  latestReady=$lr  new-rev images=$imgs (quota-blocked)"
echo "  route=[$route]  served-through-route=[$served ]"

echo "════════════════════════════════════════"
# CAPTURE (the bug): route latched to the new rev while it has FEWER than initial-scale ready pods.
if [ "$lr" = "$NEWREV" ] && [ "$maxrp" -lt "$INITIAL_SCALE" ]; then
  case "$KEEP" in oncapture|1|always) _keep=1;; esac
  echo "OUTCOME: route promoted to $NEWREV with only ${maxrp}/${INITIAL_SCALE} ready pods (Revision Ready=$newready, no image cache)."
  echo "         latestReadyRevision advanced below initial-scale and the Route follows it — the under-scaled-capture bug."
  echo "         (served-through-route=$served: a request to the under-scaled rev — 503/504/non-200 = the abandoned-capacity failure.)"
  [ "$EXPECT" = "captured" ] && { echo "PASS (expected captured)"; exit 0; } || { echo "FAIL: expected 'held' (fix regressed?)"; exit 1; }
fi

if [ "$maxrp" -ge "$INITIAL_SCALE" ]; then
  echo "OUTCOME: new rev reached full initial-scale (anti-affinity cap failed? NEWREV=$NEWREV) — under-scale not demonstrated."
else
  echo "OUTCOME: route did NOT promote to $NEWREV within ${POLL_TIMEOUT}s (latestReady=$lr); it kept serving the prior revision."
fi
[ "$EXPECT" = "held" ] && { echo "PASS (expected held — route never moved below initial-scale)"; exit 0; } || { echo "NO-REPRO"; exit 1; }

[juanpark-dandy]

[juanpark-dandy]

kserve/kserve#5683

[dprotaso]

Do you have a repro without using KServe and just vanilla Knative?