SSH Key injection issue with latest Debian image

[amingaud]

Jenkins and plugins versions report

Environment

Jenkins: 2.492.3
OS: Linux - 6.1.0-13-amd64
Java: 21.0.6 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
analysis-model-api:13.2.0
ansicolor:1.0.6
ant:513.vde9e7b_a_0da_0f
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.4.3-140.v2516ccde99e7
asm-api:9.8-135.vb_2239d08ee90
authentication-tokens:1.131.v7199556c3004
authorize-project:2.0.0
basic-branch-build-strategies:228.v68c089762a_db_
blueocean:1.27.18
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.18
blueocean-commons:1.27.18
blueocean-config:1.27.18
blueocean-core-js:1.27.18
blueocean-dashboard:1.27.18
blueocean-display-url:2.4.4
blueocean-events:1.27.18
blueocean-git-pipeline:1.27.18
blueocean-github-pipeline:1.27.18
blueocean-i18n:1.27.18
blueocean-jira:1.27.18
blueocean-jwt:1.27.18
blueocean-personalization:1.27.18
blueocean-pipeline-api-impl:1.27.18
blueocean-pipeline-editor:1.27.18
blueocean-pipeline-scm-api:1.27.18
blueocean-rest:1.27.18
blueocean-rest-impl:1.27.18
blueocean-web:1.27.18
bootstrap5-api:5.3.3-2
bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
branch-api:2.1217.v43d8b_b_d8b_2c7
build-timeout:1.38
buildtriggerbadge:251.vdf6ef853f3f5
built-on-column:1.5
caffeine-api:3.2.0-166.v72a_6d74b_870f
checks-api:370.vb_61a_c57328f3
cloud-stats:377.vd8a_6c953e98e
cloudbees-bitbucket-branch-source:936.0.1
cloudbees-folder:6.1012.v79a_86a_1ea_c1f
command-launcher:123.v37cfdc92ef67
commons-compress-api:1.27.1-3
commons-httpclient3-api:3.1-3
commons-lang3-api:3.17.0-87.v5cf526e63b_8b_
commons-text-api:1.13.0-153.v91dcd89e2a_22
config-file-provider:982.vb_a_e458a_37021
configuration-as-code:1958.vddc0d369b_e16
configuration-as-code-groovy:1.1
coverage:2.4.0
credentials:1415.v831096eb_5534
credentials-binding:687.v619cb_15e923f
data-tables-api:2.2.2-1
dependency-track:6.0.1
depgraph-view:1.0.5
display-url-api:2.209.v582ed814ff2f
docker-commons:451.vd12c371eeeb_3
docker-java-api:3.5.0-108.v211cdd21c383
docker-plugin:1274.vc0203fdf2e74
docker-workflow:611.v16e84da_6d3ff
durable-task:587.v84b_877235b_45
echarts-api:5.6.0-3
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1876.v28d8d38315b_d
envinject:2.926.v69c9b_3896a_96
envinject-api:1.235.va_14c74f8f487
external-monitor-job:223.vb_fddcf42c9b_3
favorite:2.225.v68765b_b_a_1fa_3
flatpickr-api:4.6.13-18.vcf5f6a_5b_8468
font-awesome-api:6.7.2-1
forensics-api:3.1.0
git:5.7.0
git-client:6.1.3
git-parameter:439.vb_0e46ca_14534
git-server:137.ve0060b_432302
github:1.43.0
github-api:1.321-488.v9b_c0da_9533f8
github-branch-source:1815.v9152b_2ff7a_1b_
gitlab-api:5.6.0-100.v83f8f4b_f1129
gitlab-branch-source:718.v40b_5f0e67cd3
gitlab-oauth:1.22
gitlab-plugin:1.9.8
gradle:2.14.1
groovy:497.v7b_061a_a_de65d
gson-api:2.13.0-133.v5a_e3236a_8251
h2-api:11.1.4.199-36.vb_ee07e965744
handy-uri-templates-2-api:2.1.8-36.v85e4cb_234a_13
hashicorp-vault-plugin:371.v884a_4dd60fb_6
htmlpublisher:425
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:82.v0597178874e1
jackson2-api:2.18.3-402.v74c4eb_f122b_2
jacoco:3.3.7
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-2
javadoc:327.vdfe586651ee0
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.18
jersey2-api:2.45-154.v4ded3dc34f81
jira:3.15
jira-steps:2.0.180.vccfe35b_5910d
jjwt-api:0.11.5-120.v0268cf544b_89
jnr-posix-api:3.1.20-138.vdb_9db_a_39182f
jobConfigHistory:1305.vf20a_356586b_8
joda-time-api:2.14.0-127.v7d9da_295a_d51
jquery:1.12.4-3
jquery3-api:3.7.1-3
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250107-125.v28b_a_ffa_eb_f01
json-path-api:2.9.0-148.v22a_7ffe323ce
jsoup:1.19.1-38.v216a_f3721b_3c
junit:1322.v1556dc1c59a_f
junit-attachments:299.v2e00f511b_538
kubernetes:4324.vfec199a_33512
kubernetes-client-api:6.10.0-251.v556f5f100500
kubernetes-credentials:192.v4d5b_1c429d17
ldap:780.vcb_33c9a_e4332
lockable-resources:1349.v8b_ccb_c5487f7
log-parser:2.3.7
mailer:489.vd4b_25144138f
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-auth:3.2.6
matrix-project:847.v88a_f90ff9f20
maven-plugin:3.25
metrics:4.2.30-471.v55fa_495f2b_f5
mina-sshd-api-common:2.15.0-161.vb_200831a_c15b_
mina-sshd-api-core:2.15.0-161.vb_200831a_c15b_
neoload-jenkins-plugin:2.2.11
nexus-artifact-uploader:2.14
oic-auth:4.494.v6b_f419104767
okhttp-api:4.11.0-189.v976fa_d3379d6
opentelemetry:3.1513.va_7b_d9d2324e5
opentelemetry-api:1.49.0.59.vc32561e77cf3
oss-symbols-api:324.v432cce4172ca_
pam-auth:1.12
parameterized-scheduler:285.ve611986d4c48
parameterized-trigger:859.vb_e3907a_07a_16
pipeline-build-step:567.vea_ce550ece97
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:235.vb_a_a_36b_f248c2
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:517.vf8e782ee645c
pipeline-maven:1508.v347c4b_692202
pipeline-maven-api:1508.v347c4b_692202
pipeline-milestone-step:127.vb_52887ca_3b_6d
pipeline-model-api:2.2254.v2a_978de46f35
pipeline-model-definition:2.2254.v2a_978de46f35
pipeline-model-extensions:2.2254.v2a_978de46f35
pipeline-rest-api:2.37
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2254.v2a_978de46f35
pipeline-stage-view:2.37
pipeline-utility-steps:2.19.0
plain-credentials:195.vb_906e9073dee
plugin-util-api:6.1.0
prism-api:1.30.0-1
prometheus:819.v50953a_c560dd
pubsub-light:1.19
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
rocketchatnotifier:1.5.2
run-condition:243.v3c3f94e46a_8b_
scm-api:704.v3ce5c542825a_
scmskip:72.vc2f9088d11e2
script-security:1373.vb_b_4a_a_c26fa_00
snakeyaml-api:2.3-125.v4d77857a_b_402
sonar:2.18
sse-gateway:1.28
ssh:158.ve2a_e90fb_7319
ssh-agent:384.ve275343791a_6
ssh-credentials:355.v9b_e5b_cde5003
ssh-slaves:3.1031.v72c6b_883b_869
sshd:3.353.v2b_d33c46e970
statistics-gatherer:2.0.3
structs:343.vdcf37b_a_c81d5
timestamper:1.28
token-macro:444.v52de7e9c573d
trilead-api:2.209.v0e69b_c43c245
uno-choice:2.8.7
variant:70.va_d9f17f859e0
warnings-ng:12.5.0
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1371.ve334280b_d611
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4106.v7a_8a_8176d450
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1520.v56d65e3b_4566
workflow-multibranch:806.vb_b_688f609ee9
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:700.v6e45cb_a_5a_a_21
workflow-support:968.v8f17397e87b_8
ws-cleanup:0.48

What Operating System are you using (both controller, and any agents involved in the problem)?

We are using Jenkins Controller on docker (debian 12.2 host).
Image: jenkins:2.492.3-lts-jdk21
Docker Cloud Server : Debian 10.13
Node Image: jenkins/ssh-agent:jdk21

Reproduction steps

Step 1 : set up a docker cloud on tcp, with an agent template using the image ssh-agent:jdk21.

The template has "/home/jenkins" as FS root, "Inject SSH Key" as connection method with "jenkins" as user.

Step 3 : try run remote docker agent from a job

Expected Results

Running agent.

Actual Results

Agent fails to start.
On Jenkins, we have a ssh error : SSH service hadn't started
On docker host, we have this logs :

+ [[ '' == ssh-* ]]
+ [[ '' == ssh-* ]]
+ env
+ grep _
+ [[ 8 -gt 0 ]]
+ echo 'setup-sshd params: /usr/sbin/sshd' -D -p 22 -o AuthorizedKeysCommand=/root/authorized_key -o AuthorizedKeysCommandUser=root
+ [[ /usr/sbin/sshd == ssh-* ]]
+ [[ /usr/sbin/sshd -D -p 22 -o AuthorizedKeysCommand=/root/authorized_key -o AuthorizedKeysCommandUser=root == \/\u\s\r\/\s\b\i\n\/\s\s\h\d\ \-\D\ \-\p\ \2\2 ]]
+ echo 'Executing params: '\''/usr/sbin/sshd' -D -p 22 -o AuthorizedKeysCommand=/root/authorized_key -o 'AuthorizedKeysCommandUser=root'\'''
+ exec /usr/sbin/sshd -D -p 22 -o AuthorizedKeysCommand=/root/authorized_key -o AuthorizedKeysCommandUser=root
setup-sshd params: /usr/sbin/sshd -D -p 22 -o AuthorizedKeysCommand=/root/authorized_key -o AuthorizedKeysCommandUser=root
Executing params: '/usr/sbin/sshd -D -p 22 -o AuthorizedKeysCommand=/root/authorized_key -o AuthorizedKeysCommandUser=root'
sshd: no hostkeys available -- exiting.

Anything else?

With the same configuration, there is no issue using the 6.11.1 version of ssh-agent image.
The issue is the same on jdk17 version of the image.

Are you interested in contributing a fix?

No response

[dduportal]

I can't reproduce with a standalone Docker SSH agent in Docker Compose:

services:
  jenkins-agent:
    image: jenkins/ssh-agent:jdk21
    environment:
      # This is a public key. The associated private key must be mounted in the Jenkins controller container.
      - "JENKINS_AGENT_SSH_PUBKEY=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIORe0Mi78bQ+8G3pVS4pv5VDilqHVNIhfj+UxvVD100c noone@localhost.local"

The error message sshd: no hostkeys available -- exiting. is most probably related to the security fix in https://github.com/jenkinsci/docker-ssh-agent/releases/tag/6.11.2 where we removed the SSH server-side host key embedded in the image.

This error message should not appear because the entrypoint is expected to generate a new set of SSH server-side host key during the initial startup as per

docker-ssh-agent/setup-sshd

Lines 78 to 79 in a3c3b18

	# generate host keys if not present
	ssh-keygen -A

.

=> It means the Docker Cloud is starting the SSH agent container with an unexpected set of arguments/CMD/environment variables which fails the script one way or the other.

(I'll update the issue with more analysis)

[dduportal]

Confirmed the problem is on the Docker plugin side: it should not provide the whole /usr/sbin/sshd -D -p 22 -o AuthorizedKeysCommand=/root/authorized_key -o AuthorizedKeysCommandUser=root command without the ssh-* publickey.
It has been misusing since some time as the code in

docker-ssh-agent/setup-sshd

Lines 59 to 75 in a3c3b18

	if [[ $1 == ssh-* ]]; then
	echo "Authorizing ssh pubkey found in params."
	write_key "$1"
	shift 1
	elif [[ "$@" == "/usr/sbin/sshd -D -p 22" ]]; then
	# neutralize default jenkins docker-plugin command
	# we will run sshd at the end anyway
	echo "Ignoring provided sshd command."
	# if unquoted (4 tokens) shift extra 3
	[[ "$2" == "-D" ]] && shift 3
	shift 1
	else
	echo "Executing params: '$@'"
	exec "$@"
	fi

states it expects the command to be /usr/sbin/sshd -D -p 22 (and to be "neutralized").

=> There might be a fix possible in the entrypoint to detect docker plugin usage (and neutralize it) in a less fragile way.

[dduportal]

I'll work on a fix next week. You should pin your image to 6.11.1 until then.

[MarkEWaite]

states it expects the command to be /usr/sbin/sshd -D -p 22 (and to be "neutralized").
=> There might be a fix possible in the entrypoint to detect docker plugin usage (and neutralize it) in a less fragile way.

Since I'm one of the maintainers of docker-plugin, we could also consider fixing the plugin to stop the misuse.

[amingaud]

Thank you for your quick answer. We'll work with the previous version, waiting for the fix.