ImagePullSecrets is not read by InterLink for singularity

[antoinetran]

Short Description of the issue

The image pull from a private registry that reject anonymous token fails.

Environment

• Operating System: Linux
• Other related components versions: InterLink 0.3.1-rc2 InterLink Slurm Plugin 0.3.2

Steps to reproduce

Logs, stacktrace, or other symptoms

WARNING: Cache disabled - cache location / is not writable.
INFO:    Downloading library image to tmp cache: /tmp/sbuild-tmp-cache-1796594801
FATAL:   Unable to handle docker://[some registry]/some/image uri: failed to get checksum for XXXXX: Requesting bearer token: invalid status code from registry 403 (Forbidden)

[antoinetran]

Summary of proposed changes

The Kubernetes YAML describe the imagePullSecrets, AFAIK, by two ways: either directly in pod/job description, or in serviceAccount.
Job description:

apiVersion: batch/v1
kind: Job
metadata:
  name: helloworld
spec:
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: "kubernetes.io/hostname"
                operator: In
                values:
                - interlink-slurm
      serviceAccount: serviceAccount1
      imagePullSecrets:
        - name: regcred-reg1
        - name: regcred-reg2

By serviceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: serviceAccount1
imagePullSecrets:
  - name: regcred-reg3

My suggestion (A) is that InterLink node, before sending request to InterLink Remote, does some kubectl calls to get the list of imagePullSecrets (in this example [ "regcred-reg1", "regcred-reg2", "regcred-reg3" ], then gets the triplet of { url, username, password } for each list, then optionally parses the image URL to get the registry URL domain, and does the singularity remote login (see https://apptainer.org/user-docs/master/cli/singularity_remote_login.html) .

[antoinetran]

My suggestion (B) is to add an annotation like slurm-job.vk.io/singularity-registry-username and slurm-job.vk.io/singularity-registry-password, so that InterLink does the login without parsing the Kubernetes secrets. However that means hard-coded password, which is obviously a bad think in the long-term.

[dciangot]

I definetely vote for option A. It requires:

• interlink has to add imagePullSecrets to the list here https://github.com/interTwin-eu/interLink/blob/main/pkg/interlink/api/func.go#L76
• mount the secrets in the slurm plugin https://github.com/interTwin-eu/interlink-slurm-plugin/blob/main/pkg/slurm/prepare.go#L270
• read the secrets and add singularity login as a pre-exec here https://github.com/interTwin-eu/interlink-slurm-plugin/blob/b98277004a1ad798964e620e6f274b1a1196d55b/pkg/slurm/prepare.go#L467-L481

@antoinetran does it make sense to you?

[antoinetran]

@dciangot I didn't know that InterLink Slurm plugin already had a context of secrets, so this is a good thing!
However if I understood you well, you suggest mounting the imagePullSecrets inside the singularity container. But this is already too late, instead the secrets should be read by InterLink Slurm plugin. The singularity container should have no knowledge of the imagePullSecrets so no need to mount it there.

[dciangot]

No, I mean, the prepare mount occurs before the Singularity exec. It's creating a folder that then is mounted.

I propose to use thst prepare mount to store the secret in a file and, instead of mounting it in Singularity, read the content and execute a pre-exec command. That will be run before the Container exec

[antoinetran]

The general principle seems good to me, but I would need to look more at the code.
As a workaround, I tested this and it works:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    slurm-job.vk.io/pre-exec: singularity registry login --username XXX --password XXX docker://gitlab...