[Regression 9.34.3] Package upgrade triggers cluster-wide DRBD disconnect via drbd-graceful-shutdown.service ExecStop

[iklesov]

Summary

Upgrading drbd-utils from 9.34.0-1 to 9.34.3-1 on a running production cluster causes all DRBD connections to be torn down and not re-established. The cluster ends up fragmented, with each node showing peers as Connecting and resources as Unknown/Diskless from the LINSTOR controller's perspective. Manual intervention (drbdadm adjust all on every node) is required to recover.

Environment

• drbd-utils: 9.34.3-1 (built 2026-04-17), upgraded from 9.34.0-1
• DRBD kernel module: 9.3.1
• LINSTOR: linstor-server 1.27.1, satellites on each node
• Kernel: 6.17.13-2-pve (Proxmox VE)
• Cluster: 4 nodes (pvevsan1–4), LINSTOR over ZFS_THIN on FC SAN, ~8 active resources, multiple primary VMs running

Root Cause Analysis

The package's postinst script (auto-generated by dh_systemd_start) invokes:

deb-systemd-invoke start 'drbd-configured.target' 'drbd-graceful-shutdown.service'

Under standard deb-systemd-invoke semantics, this performs a stop followed by a start for drbd-graceful-shutdown.service. The service is defined as:

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/lib/drbd/scripts/drbd-service-shim.sh down all

So during the upgrade, ExecStop runs, which executes drbdsetup down all. This:

1. Disconnects all connections for all resources (succeeds unconditionally).
2. Attempts to demote primary resources (fails with (-12) Device is held open by someone for resources used by running VMs).
3. Leaves the cluster in a state where connections are torn down but resources remain active locally — and disconnects are not rolled back when subsequent steps fail.

Reproduction Steps

1. Have a healthy multi-node DRBD cluster with active primaries (e.g., running VMs holding /dev/drbdX open).
2. Run apt upgrade drbd-utils from 9.34.0-1 to 9.34.3-1 on every node.
3. Observe: connections between nodes drop to Connecting/StandAlone and do not recover.

Evidence from Logs

/var/log/dpkg.log on pvevsan1:

2026-04-18 13:45:52 upgrade drbd-utils:amd64 9.34.0-1 9.34.3-1
2026-04-18 13:45:53 configure drbd-utils:amd64 9.34.3-1 <none>
2026-04-18 13:45:55 status installed drbd-utils:amd64 9.34.3-1

journalctl (same time window) — systemd stops the service:

Apr 18 13:45:52 pvevsan1 systemd[1]: Stopping drbd-graceful-shutdown.service - ensure all DRBD resources shut down gracefully at system shut down...

drbd-service-shim.sh output — demote fails for primaries with active VMs:

Apr 18 13:45:53 pvevsan1 drbd-service-shim.sh[2970916]: pm-47d8523a: State change failed: (-12) Device is held open by someone
Apr 18 13:45:53 pvevsan1 drbd-service-shim.sh[2970916]: failed to demote
Apr 18 13:45:53 pvevsan1 drbd-service-shim.sh[2970916]: /dev/drbd1005 open_cnt:1, writable:1; list of openers follows
Apr 18 13:45:53 pvevsan1 drbd-service-shim.sh[2970916]: drbd1005 opened by kvm (pid 2754028)

Kernel log — disconnects succeed despite demote failures:

Apr 18 13:45:53 pvevsan1 kernel: drbd pm-b425b652 pvevsan2: conn( Connected -> Disconnecting ) peer( Primary -> Unknown ) [down]
Apr 18 13:45:53 pvevsan1 kernel: drbd pm-b425b652 pvevsan2: conn( Disconnecting -> StandAlone ) [disconnected]
Apr 18 13:45:53 pvevsan1 kernel: drbd pm-b425b652 pvevsan3: Cluster is now split

The same sequence repeats on each node as it gets upgraded (pvevsan2 at 13:46:30, pvevsan3 at 13:46:56, pvevsan4 at 13:47:23).

Resulting State

After the upgrade completed on all nodes, linstor resource list showed:

• All connections in Connecting state
• Most resources reported as Unknown from peers' perspective
• Two diskless InUse resources (whose peers became unreachable) effectively isolated

drbdadm status on each node showed only the locally-active resources, with all peers stuck in Connecting.

Workaround

Run on every node:

drbdadm adjust all

This re-establishes the configuration in the kernel and reconnects peers. After applying on all four nodes, the cluster recovered fully (with normal resyncs for resources that had diverged briefly).

Expected Behavior

Upgrading the drbd-utils package on a healthy production cluster must not disrupt active DRBD connections.

Possible fixes (any of which would address the issue):

1. The drbd-graceful-shutdown.service's ExecStop should not run during package upgrades. One option is to add a guard that detects whether the system is actually shutting down (e.g., check for systemctl is-system-running or test a flag set by shutdown.target).
2. The dh_systemd_start directive in debian/rules (or equivalent) could be customized to use --no-restart-after-upgrade for drbd-graceful-shutdown.service.
3. drbd-service-shim.sh down all could be made transactional: if any disconnect succeeds but a subsequent demote/down fails, the disconnects should be rolled back.

The first or second option seems the most surgical.

Severity

This affects every multi-node DRBD cluster running active workloads at the moment of an apt upgrade drbd-utils. The window of disruption is small (10–15 minutes in our case before manual recovery) but the operation is invisible to the operator until they check linstor resource list or notice replication has stopped. For clusters with strict quorum policies, this could cause I/O errors visible to applications.

Notes

• The upgraded version (9.34.3-1) appears to be very recent (build dated 2026-04-17).
• We had no prior issues with 9.34.0-1.
• We have not yet investigated whether downgrading or rebuilding the postinst would mitigate, since the fix is needed upstream.

Acknowledgements

Log collection, root cause analysis, and the writing of this report were done with the assistance of Claude (Anthropic), an AI assistant. The reproduction, kernel/userspace logs, and verification of the workaround on a live production cluster are mine.

[chrboe]

Are you using selinux? 9.34.3 was an "emergency release" because we discovered some selinux-related bugs.

Sorry, misread the post, obviously you won't use selinux on Debian 🙂
Anyway, that release basically only contained selinux related fixes, so it seems dubious that those would break anything.

Do you have full (redacted?) syslog/journal/dmesg from these nodes? That would help in narrowing down the issue.

[iklesov]

No SELinux — confirmed, plain Debian/Proxmox. Your skepticism about the SELinux-related fixes causing this is fair, but the behavior we observed is reliably reproducible and the timing lines up exactly with the package upgrade.

Attached is drbd-incident-2026-04-18.tar.gz containing the following from all four nodes (pvevsan1–4) for the window 13:40–14:35 on 2026-04-18:

• all-pvevsanN.log — full journalctl output
• kernel-drbd-pvevsanN.log — kernel messages filtered for drbd
• linstor-sat-pvevsanN.log — linstor-satellite journal

Key timestamps to look at:

• 13:45:52–55 on pvevsan1 — drbd-utils upgrade, drbd-graceful-shutdown.service stopped by systemd, drbd-service-shim.sh down all executed, DRBD connections torn down
• 13:46:30 — same happens on pvevsan2
• 13:46:56 — same on pvevsan3
• 13:47:21 — same on pvevsan4

The specific lines that pinpoint the mechanism (from pvevsan1):
Apr 18 13:45:52 pvevsan1 systemd[1]: Stopping drbd-graceful-shutdown.service - ensure all DRBD resources shut down gracefully at system shut down...
Apr 18 13:45:53 pvevsan1 drbd-service-shim.sh[2970916]: pm-47d8523a: State change failed: (-12) Device is held open by someone
Apr 18 13:45:53 pvevsan1 drbd-service-shim.sh[2970916]: failed to demote
Apr 18 13:45:53 pvevsan1 drbd-service-shim.sh[2970916]: drbd1005 opened by kvm (pid 2754028)

The demote failure is expected (VMs hold the devices open), but the preceding disconnect from drbdsetup down all still succeeds and is not rolled back. That leaves connections torn down while resources remain active locally, and they don't reconnect on their own.

Please let me know if you need anything else (e.g., dpkg logs, specific resource .res files, or a longer time window).

drbd-incident-2026-04-18.tar.gz

[rck]

just as a quick update: confirmed. I don't think this is a regression with a recent version, I think this behavior exists since drbd-graceful-shutdown.service was introduced. For sure this is not how things should work...

In the next release this service will be split into two (for other reasons), but one of these would still show that unwanted behavior on package upgrades. I have a patch for the debian build magic so that it does not restart these services on upgrade. this will take a bit to review, but will be in 9.35.0 which should land "soon".

[iklesov]

Thanks for the quick confirmation and for working on a fix — much appreciated. Looking forward to 9.35.0.