Summary
VirtualMCPServerReconciler.deploymentNeedsUpdate never compares the pod's volumes or volumeMounts. As a result, a change to any input that only affects volumes (without also changing the container env, args, or the rendered config checksum) is silently missed: the operator reports no drift, the live pod keeps its stale mounts, and the change never takes effect until the Deployment is recreated for some unrelated reason.
This is a latent correctness gap, not an active loop. It's the inverse of #5616 (which was a false-positive drift causing a hot update loop); this is a false-negative drift.
Where
deploymentNeedsUpdate (cmd/thv-operator/controllers/virtualmcpserver_controller.go:1588) delegates to:
containerNeedsUpdate (:1636) — compares container args, env, service accountdeploymentMetadataNeedsUpdatepodTemplateMetadataNeedsUpdate — compares the runconfig-checksum annotation (hash of the rendered vMCP ConfigMap content)podTemplateSpecNeedsUpdate (:1743) — compares a hash of the user-provided spec.podTemplateSpec onlyimagePullSecretsNeedsUpdate (:1782)spec.replicas
None of these compare PodSpec.Volumes or the containers' VolumeMounts.
Impact
The build path injects several volume sets the drift check can't see:
- auth-server volumes via
GenerateAuthServerVolumes (virtualmcpserver_deployment.go:179), built from authServerConfig.SigningKeySecretRefs, HMACSecretRefs, and the Redis TLS CA cert ref
- OIDC CA-bundle volumes via
AddOIDCConfigRefCABundleVolumes (:321)
- MCPServerEntry CA-bundle volumes (
:162) and telemetry CA-bundle volumes (:172)
Each volume's SecretName/ConfigMap name comes straight from a ref, but the rendered config the checksum covers references only the mounted file paths (derived from index/name), not the backing secret/configmap name. So repointing one of those refs at a different secret (e.g. rotating SigningKeySecretRefs[0].Name from keys-v1 to keys-v2) changes the desired volume but not the checksum, and the drift check returns false. The pod keeps mounting the old secret indefinitely.
Suggested fix
Mirror the existing imagePullSecretsNeedsUpdate approach (:1782): compute a deterministic hash of the desired volume + volumeMount set, store it in an annotation on the Deployment, and compare that hash in the drift chain. A hash-annotation avoids the K8s-defaulting noise that a naive reflect.DeepEqual on live PodSpec.Volumes would trip over (defaultMode, etc.), which is the same reason imagePullSecrets and podTemplateSpec already use hashes rather than direct comparison.
Related
Summary
VirtualMCPServerReconciler.deploymentNeedsUpdatenever compares the pod's volumes or volumeMounts. As a result, a change to any input that only affects volumes (without also changing the container env, args, or the rendered config checksum) is silently missed: the operator reports no drift, the live pod keeps its stale mounts, and the change never takes effect until the Deployment is recreated for some unrelated reason.This is a latent correctness gap, not an active loop. It's the inverse of #5616 (which was a false-positive drift causing a hot update loop); this is a false-negative drift.
Where
deploymentNeedsUpdate(cmd/thv-operator/controllers/virtualmcpserver_controller.go:1588) delegates to:containerNeedsUpdate(:1636) — compares container args, env, service accountdeploymentMetadataNeedsUpdatepodTemplateMetadataNeedsUpdate— compares therunconfig-checksumannotation (hash of the rendered vMCP ConfigMap content)podTemplateSpecNeedsUpdate(:1743) — compares a hash of the user-providedspec.podTemplateSpeconlyimagePullSecretsNeedsUpdate(:1782)spec.replicasNone of these compare
PodSpec.Volumesor the containers'VolumeMounts.Impact
The build path injects several volume sets the drift check can't see:
GenerateAuthServerVolumes(virtualmcpserver_deployment.go:179), built fromauthServerConfig.SigningKeySecretRefs,HMACSecretRefs, and the Redis TLS CA cert refAddOIDCConfigRefCABundleVolumes(:321):162) and telemetry CA-bundle volumes (:172)Each volume's
SecretName/ConfigMapname comes straight from a ref, but the rendered config the checksum covers references only the mounted file paths (derived from index/name), not the backing secret/configmap name. So repointing one of those refs at a different secret (e.g. rotatingSigningKeySecretRefs[0].Namefromkeys-v1tokeys-v2) changes the desired volume but not the checksum, and the drift check returns false. The pod keeps mounting the old secret indefinitely.Suggested fix
Mirror the existing
imagePullSecretsNeedsUpdateapproach (:1782): compute a deterministic hash of the desired volume + volumeMount set, store it in an annotation on the Deployment, and compare that hash in the drift chain. A hash-annotation avoids the K8s-defaulting noise that a naivereflect.DeepEqualon livePodSpec.Volumeswould trip over (defaultMode, etc.), which is the same reason imagePullSecrets and podTemplateSpec already use hashes rather than direct comparison.Related