systemd: retry when the dbus connection returns EAGAIN by jianghao65536 · Pull Request #45 · opencontainers/cgroups

jianghao65536 · 2025-10-09T17:19:30Z

Before version 254, systemd used the glibc macro SOMAXCONN to set the backlog. Prior to Linux 5.4, SOMAXCONN was set to 128. If the number of instantaneous connections exceeds this value, newConnection will return EAGAIN, which can cause runc to fail when creating containers.

kolyshkin

I tried the test from here (without the fix, just the test case) on three different distros:

Fedora 42: kernel 6.16.10-200.fc42.x86_64, systemd 257.9-2.fc42
Alma 9: kernel 5.14.0-570.37.1.el9_6.x86_64, systemd 252-51.el9.alma.1
Alma 8: kernel 4.18.0-553.16.1.el8_10.x86_64, systemd 239-82.el8_10.1

and it does not fail.

This might be because Red Hat has backported systemd fix, so what distro do you recommend to try so we can see the test fails before the fix?

jianghao65536 · 2025-10-26T06:04:06Z

@kolyshkin I can reproduce the issue using Alma 8. The reproducible Vagrantfile is as follows:

Vagrant.configure("2") do |config|
  config.vm.box = "almalinux/8"
  config.vm.hostname = "alma8-vm"

  config.vm.provider "virtualbox" do |vb|
    vb.name = "AlmaLinux8_Vagrant"
    vb.memory = "2048"
    vb.cpus = 2
  end

  config.vm.provision "shell", inline: <<-SHELL
    sudo dnf install -y go
    go version
  SHELL
end

The test result is as follows:

I haven’t tested it on many kernel versions, but based on the commit messages of systemd and glibc, it seems that systemd compiled with glibc versions earlier than 2.31 may encounter the EAGAIN issue.
Systemd related link: systemd/systemd#28018
Glibc related link: bminor/glibc@96958e2

cyphar · 2025-10-28T05:16:51Z

+			// despite retries, still EAGAIN
+			return nil, fmt.Errorf("dbus connection failed after 5 retries: %w", err)


This logic is flipped in a slightly confusing way, I'm going to push a more straightforward version.

cyphar · 2025-10-28T05:33:50Z

+		}
+		if retry < 4 {
+			// Exponential backoff with jitter: base delay doubles each retry, plus random jitter up to 1 second
+			baseDelay := time.Second << retry // 1s, 2s, 4s, 8s


15 seconds with the first step being 1s seems a little aggressive -- I would expect the first retry to be more like 100ms or something. runc generally runs on the order of 100ms so I would not expect the base delay to be this long.

I think that 100ms * 2^retry + 12.5% jitter is more reasonable. I'm not even sure the jitter is necessary, but I've kept it.

cyphar · 2025-10-28T05:41:17Z

+	"testing"
+)
+
+func TestParallelConnection(t *testing.T) {


Funnily enough, this test actually crashes Firefox on my machine if you run it in rootless mode as your user:

[1522754.178417] [ T3250] wireplumber[3250]: segfault at 8 ip 00007f8b0097315d sp 00007ffd1e267fe8 error 4 in libgobject-2.0.so.0.8600.0[3d15d,7f8b00946000+36000] likely on CPU 14 (core 7, socket 0) [1522754.178426] [ T3250] Code: 0f 1f 44 00 00 48 c1 ed 02 48 8d 05 5d 62 02 00 48 8b 34 e8 e9 48 ff ff ff 0f 1f 40 00 48 85 ff 74 4b 48 8b 07 48 85 c0 74 43 <48> 8b 00 48 3d fc 03 00 00 77 28 48 8d 15 31 62 02 00 48 c1 e8 02

Signed-off-by: jianghao65536 <jianghao65536@gmail.com> [cyphar: gofumpt systemd/dbus_test.go] [cyphar: simplify retry loop to return from inside loop] [cyphar: improve exponential backoff to be less aggressive] [cyphar: improve parallel test] Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

cyphar · 2025-10-28T06:06:45Z

+	var wg sync.WaitGroup
+	errChan := make(chan error, len(dms))
+	for _, dm := range dms {
+		wg.Add(1)
+		go func(dm *dbusConnManager) {
+			defer wg.Done()
+			select {
+			case <-ctx.Done():
+				return
+			default:
+				_, err := dm.newConnection()
+				if err != nil {
+					errChan <- err
+					cancel()
+				}
+			}
+		}(dm)
+	}
+	wg.Wait()
+	close(errChan)
+	for err := range errChan {
+		t.Fatal(err)
+	}


This doesn't actually test all connections being created at the same time, and the usage of errChan is a little suspect. I will push a different version.

Suggestion implemented.

cyphar · 2025-10-28T06:26:32Z

@kolyshkin let me know if you're happy with the new backoff setup I pushed.

kolyshkin

LGTM

cgroups: provide iocost statistics for cgroupv2. (opencontainers/cgroups#43) cgroups: retry DBus connection when it fails with EAGAIN.(opencontainers/cgroups#45) cgroups: improve cpuacct.usage_all resilience when parsing data from patched kernels (such as the Tencent kernels). (opencontainers/cgroups#46,opencontainers/cgroups#50) libct: close child fds on prepareCgroupFD error. (openwrt#4936) libct: fix mips compilation. (openwrt#4962, openwrt#4967) When configuring a tmpfs mount, only set the mode= argument if the target path already existed. This fixes a regression introduced in our CVE-2025-52881 mitigation patches. (openwrt#4971, openwrt#4976) Fix various file descriptor leaks and add additional tests to detect them as comprehensively as possible. (openwrt#5007, openwrt#5021, openwrt#5034) The "hallucination" helpers added as part of the CVE-2025-52881 mitigation have been made more generic and now apply to all of our pathrs helper functions, which should ensure we will not regress dangling symlink users. (openwrt#4985) Signed-off-by: David Mandy <smallprogramzhusir@gmail.com> Signed-off-by: David Mandy <smallprogramzhusir@gmail.com>

cgroups: provide iocost statistics for cgroupv2. (opencontainers/cgroups#43) cgroups: retry DBus connection when it fails with EAGAIN.(opencontainers/cgroups#45) cgroups: improve cpuacct.usage_all resilience when parsing data from patched kernels (such as the Tencent kernels). (opencontainers/cgroups#46,opencontainers/cgroups#50) libct: close child fds on prepareCgroupFD error. (openwrt#4936) libct: fix mips compilation. (openwrt#4962, openwrt#4967) When configuring a tmpfs mount, only set the mode= argument if the target path already existed. This fixes a regression introduced in our CVE-2025-52881 mitigation patches. (openwrt#4971, openwrt#4976) Fix various file descriptor leaks and add additional tests to detect them as comprehensively as possible. (openwrt#5007, openwrt#5021, openwrt#5034) The "hallucination" helpers added as part of the CVE-2025-52881 mitigation have been made more generic and now apply to all of our pathrs helper functions, which should ensure we will not regress dangling symlink users. (openwrt#4985) Signed-off-by: David Mandy <smallprogramzhusir@gmail.com>

cgroups: provide iocost statistics for cgroupv2. (opencontainers/cgroups#43) cgroups: retry DBus connection when it fails with EAGAIN.(opencontainers/cgroups#45) cgroups: improve cpuacct.usage_all resilience when parsing data from patched kernels (such as the Tencent kernels). (opencontainers/cgroups#46,opencontainers/cgroups#50) libct: close child fds on prepareCgroupFD error. (opencontainers/runc#4936) libct: fix mips compilation. (opencontainers/runc#4962, opencontainers/runc#4967) When configuring a tmpfs mount, only set the mode= argument if the target path already existed. This fixes a regression introduced in our CVE-2025-52881 mitigation patches. (opencontainers/runc#4971, opencontainers/runc#4976) Fix various file descriptor leaks and add additional tests to detect them as comprehensively as possible. (opencontainers/runc#5007, opencontainers/runc#5021, opencontainers/runc#5034) The "hallucination" helpers added as part of the CVE-2025-52881 mitigation have been made more generic and now apply to all of our pathrs helper functions, which should ensure we will not regress dangling symlink users. (opencontainers/runc#4985) Signed-off-by: David Mandy <smallprogramzhusir@gmail.com>

jianghao65536 force-pushed the eagain-retry branch 2 times, most recently from 350f6fc to c391dec Compare October 9, 2025 17:25

lifubang previously requested changes Oct 18, 2025

View reviewed changes

Comment thread systemd/dbus_test.go

kolyshkin reviewed Oct 24, 2025

View reviewed changes

Comment thread systemd/dbus_test.go Outdated

kolyshkin reviewed Oct 24, 2025

View reviewed changes

jianghao65536 force-pushed the eagain-retry branch from c391dec to 6db5fc0 Compare October 26, 2025 04:50

AkihiroSuda approved these changes Oct 28, 2025

View reviewed changes

AkihiroSuda added this to the 0.0.6 milestone Oct 28, 2025

cyphar reviewed Oct 28, 2025

View reviewed changes

cyphar force-pushed the eagain-retry branch from 6db5fc0 to e50ca29 Compare October 28, 2025 05:34

cyphar reviewed Oct 28, 2025

View reviewed changes

cyphar force-pushed the eagain-retry branch from e50ca29 to cd71e92 Compare October 28, 2025 06:07

cyphar approved these changes Oct 28, 2025

View reviewed changes

kolyshkin approved these changes Oct 29, 2025

View reviewed changes

kolyshkin merged commit c63eee3 into opencontainers:main Oct 29, 2025
14 checks passed

lifubang mentioned this pull request Nov 26, 2025

[1.4] Release 1.4.0 opencontainers/runc#5046

Merged

smallprogram mentioned this pull request Jan 29, 2026

runc: bump to 1.4.0 openwrt/packages#28453

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

systemd: retry when the dbus connection returns EAGAIN#45

systemd: retry when the dbus connection returns EAGAIN#45
kolyshkin merged 1 commit intoopencontainers:mainfrom
jianghao65536:eagain-retry

jianghao65536 commented Oct 9, 2025

Uh oh!

Uh oh!

Uh oh!

kolyshkin left a comment

Uh oh!

jianghao65536 commented Oct 26, 2025

Uh oh!

cyphar Oct 28, 2025

Uh oh!

cyphar Oct 28, 2025

Uh oh!

cyphar Oct 28, 2025 •

edited

Loading

Uh oh!

cyphar Oct 28, 2025 •

edited

Loading

Uh oh!

cyphar commented Oct 28, 2025

Uh oh!

kolyshkin left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		// despite retries, still EAGAIN
		return nil, fmt.Errorf("dbus connection failed after 5 retries: %w", err)

Conversation

jianghao65536 commented Oct 9, 2025

Uh oh!

Uh oh!

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

jianghao65536 commented Oct 26, 2025

Uh oh!

cyphar Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

cyphar Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

cyphar Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cyphar Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cyphar commented Oct 28, 2025

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cyphar Oct 28, 2025 •

edited

Loading

cyphar Oct 28, 2025 •

edited

Loading