[Bug]: Slurmctld, slurm-restapi and slurmd(nodesets) images does not support user identity resolution from SSSD natively

[sanketcoredge]

Description

The Slinky login image (ghcr.io/slinkyproject/login) natively bundles SSSD and exposes a first-class sssdConf helm value that allows operators to configure LDAP-based user identity resolution out of the box. However, the slurmctld and slurmd images do not include SSSD or any equivalent NSS-to-LDAP bridge.

This creates a gap when running per-user jobs with LDAP-backed POSIX identities:

slurmd calls setuid(uid) / setgid(gid) to run jobs as the mapped user. Without NSS-to-LDAP resolution, UID/GID lookups fail, $HOME and $USER are incorrect, and job execution under the correct user identity is broken.
slurmctld needs UID resolution for prolog/epilog scripts and accounting.

Steps to Reproduce

• Deploy a Slurm
• Provision a POSIX user in OpenLDAP (e.g., uid=alice, uidNumber=10001)
• Configure sssdConf on the loginset to point to the OpenLDAP instance — user resolves correctly on the login pod
• Submit a job as alice via the Slurm REST API
• Observe that Job is in PENDING state. Reason is Job Held Admin.

Environment:

slurm-operator: v1.0.0
Slurm: 25.11
Images: ghcr.io/slinkyproject/slurmctld:25.11-ubuntu24.04, ghcr.io/slinkyproject/slurmd:25.11-ubuntu24.04

Expected Behavior

Slurmctld and slurmd should have native support to resolve user identity via SSSD.

[vivian-hafener]

Good afternoon @sanketcoredge,

Could you provide more details on what you are doing for this step:

Submit a job as alice via the Slurm REST API

As well as provide logs from slurmctld and the slurm-restapi pod surrounding this event, so that I can better understand what is occurring on your cluster?

How are you submitting the job? Directly to the Slurm REST API using a POST request? Or via Slurm-operator?

Best regards,
Vivian Hafener

[sanketcoredge]

Hi @vivian-hafener ,

I have deployed openldap on a cluster where slurm is being deployed.

Also, created a configmap for sssd.conf in the cluster

Here are changes i did in slurm values.yaml file

Controller:
...
extraConf: |
    PrivateData=jobs,usage,accounts,users
    AccountingStorageEnforce=associations
...
loginsets:
    slinky: 
        enabled: true
    ...
    sssdConf: |
      [sssd]
      config_file_version = 2
      services = nss,pam
      domains = DEFAULT
      [nss]
      filter_groups = root,slurm
      filter_users = root,slurm

      [pam]

      [domain/DEFAULT]
      auth_provider = ldap
      id_provider = ldap
      ldap_uri = ldap://openldap.slurm.svc.cluster.local
      ldap_search_base = dc=compass,dc=slurm
      ldap_user_search_base = ou=Users,dc=compass,dc=slurm
      ldap_group_search_base = ou=Groups,dc=compass,dc=slurm
      ldap_default_bind_dn = cn=admin,dc=compass,dc=slurm
      ldap_default_authtok_type = password
      ldap_default_authtok = adminpassword
      enumerate = true
      cache_credentials = true

nodesets:
...
...
        volumeMounts:
        - name: sssd-config
          mountPath: /etc/sssd/sssd.conf
          subPath: sssd.conf
...
...
        volumes:
        - name: sssd-config
          configMap:
            name: sssd-config

• Created alice user in ldap.

• Added user and account for alice user in slurm accounting
  sacctmgr -i add account slurmact description="test" organization="test"
sacctmgr -i add user alice account=slurmact

• Generated JWT token for alice user and submitted job from test nginx pod using curl.

  curl -s -X POST   http://slurm-restapi.slurm.svc.cluster.local:6820/slurm/v0.0.44/job/submit --header 'X-SLURM-USER-NAME: alice' --header 'X-SLURM-USER-TOKEN: $TOKEN' --header 'Content-Type: application/json' --data '{
  "job": {
      "script": "#!/bin/bash\n\necho \"Allocating 120 MB using dd...\"\nsleep 60",
      "environment": [
          "PATH=/bin/:/usr/bin/:/sbin/"
      ],
      "name": "j5",
      "current_working_directory": "/tmp",
      "partition": "all",
      "account": "slurmact"
  }
  }'
  

  It returns this response:

{
  "errors": [
    {
      "description": "Batch job submission failed",
      "error_number": 2045,
      "error": "Invalid account or account\/partition combination specified",
      "source": "slurm_submit_batch_job()"
    }
  ],
  "warnings": [],
  "meta": {
    "plugin": {
      "type": "openapi\/slurmctld",
      "name": "Slurm OpenAPI slurmctld",
      "data_parser": "data_parser\/v0.0.44",
      "accounting_storage": "accounting_storage\/slurmdbd"
    },
    "client": {
      "source": "slurm-restapi-cbcd9cb44-85qvm:6820(fd:11)",
      "user": "root",
      "group": "root"
    },
    "command": [],
    "slurm": {
      "version": {
        "major": "25",
        "micro": "0",
        "minor": "11"
      },
      "release": "25.11.0",
      "cluster": "slurm_slurm"
    }
  }
}

• Here are logs for slurm-restapi:
  [2026-02-19T12:29:09.963] rest_auth/jwt: slurm_rest_auth_p_authenticate: [slurm-restapi-cbcd9cb44-85qvm:6820(fd:9)] attempting token authentication pass through [2026-02-19T12:29:09.964] operations_router: [slurm-restapi-cbcd9cb44-85qvm:6820(fd:12)] GET /slurm/v0.0.44/jobs/ [2026-02-19T12:29:09.965] rest_auth/jwt: slurm_rest_auth_p_authenticate: [slurm-restapi-cbcd9cb44-85qvm:6820(fd:12)] attempting token authentication pass through [2026-02-19T12:29:23.487] operations_router: [slurm-restapi-cbcd9cb44-85qvm:6820(fd:11)] POST /slurm/v0.0.44/job/submit [2026-02-19T12:29:23.487] rest_auth/jwt: slurm_rest_auth_p_authenticate: [slurm-restapi-cbcd9cb44-85qvm:6820(fd:11)] attempting user_name ccsadmin token authentication pass through [2026-02-19T12:29:23.489] error: slurm_submit_batch_job(): [slurm-restapi-cbcd9cb44-85qvm:6820(fd:11)] parser=data_parser/v0.0.44 rc[2045]=Invalid account or account/partition combination specified -> Batch job submission failed [2026-02-19T12:29:23.490] error: wrap_on_data: [slurm-restapi-cbcd9cb44-85qvm:6820(fd:11)] on_data returned rc: Invalid account or account/partition combination specified [2026-02-19T12:29:39.963] operations_router: [slurm-restapi-cbcd9cb44-85qvm:6820(fd:9)] GET /slurm/v0.0.44/nodes/ [2026-02-19T12:29:39.963] rest_auth/jwt: slurm_rest_auth_p_authenticate: [slurm-restapi-cbcd9cb44-85qvm:6820(fd:9)] attempting token authentication pass through

• Here are logs for slurmctld (Nothing significant):
  [2026-02-19T11:40:36] Running as primary controller [2026-02-19T11:40:36] child started successfully [2026-02-19T11:40:36] Relinquishing control to new slurmctld process [2026-02-19T11:40:36] error: xsystemd_change_mainpid: connect() failed for /tmp/fakesystemd.sock: No such file or directory [2026-02-19T11:40:36] Relinquishing control to new child 2026-02-19 11:40:37,495 INFO exited: slurmctld (exit status 0; expected) 2026-02-19 11:40:37,496 INFO reaped unknown pid 18 (exit status 0)

• I am able to reach to openldap service from slurm-restapi, slurm controller and slurmd pods. But when i try to resolve user getent passwd alice, i get nothing.

[vivian-hafener]

Hi @sanketcoredge,

Thanks for the logs. They've been helpful to confirm what's failing here.

This is related to some ongoing development work being undertaken by the Slinky team, related to these PRs:

• Add SSSD support to controller #93
• Add SSSD support to slurmctld container containers#8

Refactoring these portions of our codebase to do this right will take some time. It is likely that these changes, which will enable the addition of SSSD to these components, will not land before v1.2.

Best regards,
Vivian Hafener