Empty kubernetes token randomly

[antoinetran]

Issue:

Randomly (once per day?), the pod ask for a Kubernetes token (of the service account), as a projectedVolumeMap. The token is empty:

[trana@XX]$ cat projectedVolumeMaps/kube-api-access-dpc79/token
[trana@XX]$ od -a projectedVolumeMaps/kube-api-access-dpc79/token
0000000

Expected result:

The token is not empty. Eg (shortened for readiness):

[trana@XX]$ cat projectedVolumeMaps/kube-api-access-dpc79/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IjBqXzRNTWEyN3FVTzM1aW5............................................

How to reproduce:

Probably run lots lots of pods with a projectedVolumeMap:

apiVersion: v1
kind: Pod
metadata:
  name: test-token
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - arm64
            - amd64
          - key: kubernetes.io/hostname
            operator: In
            values:
            - interlink-slurm-node
        weight: 1
  initContainers:
  - command:
    - /bin/sh
    - -c
    - |
      find /var/run/secrets/kubernetes.io/serviceaccount
      ls -al /var/run/secrets/kubernetes.io/serviceaccount
      cat /var/run/secrets/kubernetes.io/serviceaccount/token
    image: alpine
    name: init
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 16Mi
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-j5rc5
      readOnly: true
  restartPolicy: Never
  serviceAccount: default
  serviceAccountName: default
  tolerations:
  - key: virtual-node.interlink/no-schedule
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-j5rc5
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token

The other volumes, including ca.crt was correctly filled. Only token was empty.

[antoinetran]

Logs:
VK:

time="2025-09-08T08:23:48Z" level=info msg="receive GetPod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:48Z" level=info msg="receive CreatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:48Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:53Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:53Z" level=debug msg="empty dir found, nothing to do for volume var-run-argo for Pod sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760"
time="2025-09-08T08:23:53Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:53Z" level=debug msg="empty dir found, nothing to do for volume tmp-dir-argo for Pod sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760"
time="2025-09-08T08:23:53Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""                                                                                                           time="2025-09-08T08:23:53Z" level=debug msg="empty dir found, nothing to do for volume tmpdir for Pod sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760"
time="2025-09-08T08:23:53Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:53Z" level=debug msg="sync handled" key="{argo-ddpc-cusim sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760 0dabe6f5-95f0-42dd-b063-2d4fa6f78c70}"
time="2025-09-08T08:23:53Z" level=debug msg="sync handled" key="{argo-ddpc-cusim sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760 0dabe6f5-95f0-42dd-b063-2d4fa6f78c70}"
time="2025-09-08T08:23:53Z" level=info msg="receive GetPod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:53Z" level=info msg="receive DeletePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:53Z" level=info msg="receive DeletePod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:53Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:53Z" level=debug msg="Not enqueuing pod status update" error="Pod {\"argo-ddpc-cusim\" \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" \"0dabe6f5-95f0-42dd-b063-2d4fa6f78c70\"} not found in pod li
ster: pod \"argo-ddpc-cusim/sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found: pod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found"
time="2025-09-08T08:23:55Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:55Z" level=debug msg="Not enqueuing pod status update" error="Pod {\"argo-ddpc-cusim\" \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" \"0dabe6f5-95f0-42dd-b063-2d4fa6f78c70\"} not found in pod li
ster: pod \"argo-ddpc-cusim/sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found: pod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found"
time="2025-09-08T08:23:57Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:57Z" level=debug msg="Not enqueuing pod status update" error="Pod {\"argo-ddpc-cusim\" \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" \"0dabe6f5-95f0-42dd-b063-2d4fa6f78c70\"} not found in pod li
ster: pod \"argo-ddpc-cusim/sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found: pod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found"
time="2025-09-08T08:23:57Z" level=debug msg="empty dir found, nothing to do for volume input-artifacts for Pod sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760"                                                                time="2025-09-08T08:23:57Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:57Z" level=debug msg="Not enqueuing pod status update" error="Pod {\"argo-ddpc-cusim\" \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" \"0dabe6f5-95f0-42dd-b063-2d4fa6f78c70\"} not found in pod li
ster: pod \"argo-ddpc-cusim/sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found: pod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found"
time="2025-09-08T08:23:57Z" level=debug msg="empty dir found, nothing to do for volume argo-staging for Pod sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760"
time="2025-09-08T08:23:57Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:57Z" level=debug msg="Not enqueuing pod status update" error="Pod {\"argo-ddpc-cusim\" \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" \"0dabe6f5-95f0-42dd-b063-2d4fa6f78c70\"} not found in pod li
ster: pod \"argo-ddpc-cusim/sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found: pod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found"
time="2025-09-08T08:23:59Z" level=error msg="error during token request in RemoteExecution() pods \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found"
time="2025-09-08T08:23:59Z" level=info msg="receive UpdatePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
time="2025-09-08T08:23:59Z" level=debug msg="Not enqueuing pod status update" error="Pod {\"argo-ddpc-cusim\" \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" \"0dabe6f5-95f0-42dd-b063-2d4fa6f78c70\"} not found in pod lister: pod \"argo-ddpc-cusim/sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found: pod \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found"
time="2025-09-08T08:23:59Z" level=debug msg="in addKubernetesServicesEnvVars InterLink VK environment variable to pod sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760 container: init env: ARGO_POD_NAME value: sim-generate-so
urce-template-r4lqm-localstorage-to-globalstorage-937119760"

So the token could not be generated because the pod was being deleted:

time="2025-09-08T08:23:53Z" level=info msg="receive DeletePodVK \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\""
....
time="2025-09-08T08:23:59Z" level=error msg="error during token request in RemoteExecution() pods \"sim-generate-source-template-r4lqm-localstorage-to-globalstorage-937119760\" not found"

[antoinetran]

It seems interlink VK receive the pod delete request. However, I do not see in events an eviction pod. So I don't know why kubernetes tries to delete the pod.