bug: vmcluster parallel rollout of components

[dctrwatson]

vmoperator v0.65.0

I just applied change to upgrade the image tags for all 3 components and also added the GOMAXPROCS to extraEnvs. Instead of waiting for vmstorage, then vmselect to complete their rollout, all 3 were updated in parallel instead.

Example VMCluster CR:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  name: canary
  namespace: monitoring
spec:
  replicationFactor: 2
  retentionPeriod: "3"
  vminsert:
    extraEnvs:
      - name: GOMAXPROCS
        value: "2"
    hpa:
      maxReplicas: 12
      metrics:
        - pods:
            metric:
              name: vm_concurrent_insert_utilization
            target:
              averageValue: 800m
              type: AverageValue
          type: Pods
      minReplicas: 6
    image:
      tag: v1.131.0-cluster
    minReadySeconds: 180
    podDisruptionBudget:
      maxUnavailable: 1
    port: "8480"
    priorityClassName: monitoring-canary
    replicaCount: 6
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 500m
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    serviceSpec:
      metadata:
        annotations:
          service.kubernetes.io/topology-mode: Auto
        name: vminsert-canary-az-aware
      spec: {}
    terminationGracePeriodSeconds: 300
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
  vmselect:
    cacheMountPath: /select-cache
    clusterNativeListenPort: "8401"
    extraArgs:
      dedup.minScrapeInterval: 60s
    extraEnvs:
      - name: GOMAXPROCS
        value: "4"
    hpa:
      maxReplicas: 6
      metrics:
        - pods:
            metric:
              name: vm_concurrent_select_utilization
            target:
              averageValue: 800m
              type: AverageValue
          type: Pods
      minReplicas: 3
    image:
      tag: v1.131.0-cluster
    minReadySeconds: 30
    podDisruptionBudget:
      maxUnavailable: 1
    port: "8481"
    priorityClassName: monitoring-canary
    replicaCount: 2
    resources:
      limits:
        memory: 3Gi
      requests:
        cpu: 1500m
    rollingUpdateStrategy: RollingUpdate
    serviceSpec:
      metadata:
        annotations:
          service.kubernetes.io/topology-mode: Auto
        name: vmselect-canary-az-aware
      spec: {}
    storage:
      emptyDir: {}
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 2Gi
    terminationGracePeriodSeconds: 60
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
  vmstorage:
    extraArgs:
      dedup.minScrapeInterval: 60s
    extraEnvs:
      - name: GOMAXPROCS
        value: "8"
    image:
      tag: v1.131.0-cluster
    minReadySeconds: 180
    podDisruptionBudget:
      maxUnavailable: 1
    priorityClassName: canary-monitoring
    replicaCount: 6
    resources:
      requests:
        cpu: '7'
      limits:
        memory: 27Gi
    rollingUpdateStrategy: RollingUpdate
    storage:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 500Gi
          storageClassName: hdd
    storageDataPath: /vm-data
    terminationGracePeriodSeconds: 900
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule

[tiny-pangolin]

setting maxUnavailable: 100% will allow restarting all instances of a component at once but there will be a small amount of downtime. If you need an ha pair of clusters look at the distributed helm chart

[f41gh7]

I just applied change to upgrade the tags for all 3 nodes and also added the GOMAXPROCS to extraEnvs. Instead of waiting for each node to complete their rollout, all 3 were updated in parallel.

Could you please a bit more specific? Which node tags were changed ( Kubernetes Node)? And WDYM by all 3 were updated in parallel ( 3 storage pods updated at once)?

I only can guess, that it's related to the podDisruptionBudget and maxUnavailable: 1.

Also, with rollingUpdateStrategy: RollingUpdate, operator doesn't perform any pod level actions, it delegates update process to the kubernetes-controller-manager.

[dctrwatson]

Could you please a bit more specific? Which node tags were changed ( Kubernetes Node)? And WDYM by all 3 were updated in parallel ( 3 storage pods updated at once)?

Sorry, vmselect, vminsert, and vmstorage components all were updated in parallel.

Also, with rollingUpdateStrategy: RollingUpdate, operator doesn't perform any pod level actions, it delegates update process to the kubernetes-controller-manager.

Yup, the operator updated the vmstorage and vmselect STS and the vminsert deployment at the same-ish time. Rather than doing them serially.

[f41gh7]

@AndrewChubatiuk @vrutkovs Could you please take a look? A was not able to reproduce it. Most probably, Kubernetes controller-manager doesn't properly reflect statefulSet.status updates during rollout. And operator relies on it with rollingUpdateStrategy: RollingUpdate.

[dctrwatson]

Started rolling out an image only change this week and so far have this only happened once out of 6 vmcluster objects so far.

I didn't notice it until about 15 minutes after it happened.
Here's the status of the vmstorage STS:

status:
  availableReplicas: 5
  collisionCount: 0
  currentReplicas: 2
  currentRevision: vmstorage-internal-686c76f5b8
  observedGeneration: 10
  readyReplicas: 5
  replicas: 6
  updateRevision: vmstorage-internal-545b68688f
  updatedReplicas: 3

The vmoperator only had this when it udpated the vmselect STS (with the diff showing the image change):

logger":"controller.VMCluster","msg":"updating statefulset vmselect-internal configuration, is_current_equal=false,is_prev_equal=false,is_prev_nil=false

[vrutkovs]

Yup, the operator updated the vmstorage and vmselect STS and the vminsert deployment at the same-ish time. Rather than doing them serially.

So the expectation is that we first update vminsert, wait for the rollout to complete, afterwards vmstorage is picked up, we wait for it to complete and only then vmselect sts is updated?

Why would starting the upgrade simultaneously be a problem if they are already backed up by multiple replicas?

[dctrwatson]

So the expectation is that we first update vminsert, wait for the rollout to complete, afterwards vmstorage is picked up, we wait for it to complete and only then vmselect sts is updated?

Usual behavior follows the No downtime strategy

Why would starting the upgrade simultaneously be a problem if they are already backed up by multiple replicas?

When all 3 roll simultaneously, there's extra load on the vmstorage tier that we don't normally see. Mostly causing prolonged query timeouts from vmalert.