Use systemd-confext instead of custom /etc overlay mount

For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>

Author: pothos

build_library/build_image_util.sh

@@ -748,12 +748,35 @@ EOF
   if [[ $(sudo find "${root_fs_dir}/usr/share/flatcar/etc" -size +0 ! -type d 2>/dev/null | wc -l) -gt 0 ]]; then
     die "Unexpected non-empty files in ${root_fs_dir}/usr/share/flatcar/etc"
   fi
+  # Some backwards-compat symlinks still use this folder as target,
+  # we can't remove it yet
   sudo rm -rf "${root_fs_dir}/usr/share/flatcar/etc"
   sudo cp -a "${root_fs_dir}/etc" "${root_fs_dir}/usr/share/flatcar/etc"
-
-  # Remove the rootfs state as it should be recreated through the
-  # tmpfiles and may not be present on updating machines. This
-  # makes sure our tests cover the case of missing files in the
+  # Now set up a default confext and enable it.
+  # It's important to use dm-verity not only for stricter image policies
+  # but also because it allows us the refresh to identify this image and
+  # skip setting it up again in the final boot, which not only saves us
+  # a daemon-reload during boot but also from /etc contents shortly
+  # disappearing until systemd-sysext uses mount beneath for an atomic
+  # remount. Instead of a temporary directory we first prepare it as
+  # folder and then convert it to a DDI and remove the folder.
+  sudo rm -rf "${root_fs_dir}/usr/lib/confexts/00-flatcar-default"
+  sudo mkdir -p "${root_fs_dir}/usr/lib/confexts/00-flatcar-default"
+  # Do a copy because we keep /etc for the flatcar (.tar) container and the developer container
+  sudo cp -a "${root_fs_dir}/etc" "${root_fs_dir}/usr/lib/confexts/00-flatcar-default/etc"
+  sudo mkdir -p "${root_fs_dir}/usr/lib/confexts/00-flatcar-default/etc/extension-release.d/"
+  echo ID=_any | sudo tee "${root_fs_dir}/usr/lib/confexts/00-flatcar-default/etc/extension-release.d/extension-release.00-flatcar-default" > /dev/null
+  sudo systemd-repart \
+    --private-key="${SYSEXT_SIGNING_KEY_DIR}/sysexts.key" \
+    --certificate="${SYSEXT_SIGNING_KEY_DIR}/sysexts.crt" \
+    --make-ddi=confext \
+    --copy-source="${root_fs_dir}/usr/lib/confexts/00-flatcar-default" \
+    "${root_fs_dir}/usr/lib/confexts/00-flatcar-default.raw"
+  sudo rm -rf "${root_fs_dir}/usr/lib/confexts/00-flatcar-default"
+
+  # Remove the rootfs state as it should be recreated through tmpfiles
+  # (and for /etc we use a confext) and may not be present on updating machines.
+  # This makes sure our tests cover the case of missing files in the
   # rootfs and don't rely on the new image. Not done for the developer
   # container.
   if [[ -n "${image_kernel}" ]]; then

changelog/changes/2025-12-12-default-systemd-confext.md

@@ -0,0 +1,2 @@
+- Switched `/etc/` from a custom overlayfs for A/B updates to using a systemd-confext extension providing the default contents by using systemd-confext in the mutable mode where `/etc/` gets used as upperdir [scripts#3555](https://github.com/flatcar/scripts/pull/3555)
+- Moved systemd-sysext image mounting into the initrd, so that system extensions can better define the behavior of the final system at boot without workarounds to apply settings late at boot. This means `.wants` symlinks for systemd units work as expected now and, therefore, we dropped the `ensure-sysext.service` workaround. We still recommend extensions to keep their workarounds, e.g., using `.upholds` instead of `.wants`, to better support live reloading. A skipping logic prevents an extension refresh late at boot but only if no changes were found. For extensions that are not stored on a custom filesystem, such as a separate `/var` partition, the new extension mounting from the initrd won't be able to load them early but they will be picked up late at boot through the extension refresh. This is another case where it's good if extensions keep workarounds for late loading.

sdk_container/src/third_party/coreos-overlay/coreos-base/coreos-init/coreos-init-9999.ebuild

@@ -8,7 +8,8 @@ EGIT_REPO_URI="https://github.com/flatcar/init.git"
 if [[ "${PV}" == 9999 ]]; then
 	KEYWORDS="~amd64 ~arm ~arm64 ~x86"
 else
-	EGIT_COMMIT="8bd8a82fb22bc46ea2cf7da94d58655e102ca26d" # flatcar-master
+	#EGIT_COMMIT="8bd8a82fb22bc46ea2cf7da94d58655e102ca26d" # flatcar-master
+	EGIT_BRANCH="kai/default-confext"
 	KEYWORDS="amd64 arm arm64 x86"
 fi
 

sdk_container/src/third_party/coreos-overlay/coreos/config/env/sys-apps/systemd

@@ -160,19 +160,6 @@ EOF
         -e '/^C!* \/etc\/pam\.d/d' \
         -e '/^C!* \/etc\/issue/d' || die
 
-    (
-        # Some OEMs prefer chronyd, so allow them to replace
-        # systemd-timesyncd with it.
-        insinto "$(systemd_get_systemunitdir)/systemd-timesyncd.service.d"
-        newins - flatcar.conf <<'EOF'
-# Allow sysexts to ship timesyncd replacements which can have
-# a Conflicts=systemd-timesyncd directive that would result
-# in systemd-timesyncd not being started.
-[Unit]
-After=ensure-sysext.service
-EOF
-    )
-
     (
         # Allow @mount syscalls for systemd-udevd.service
         insinto "$(systemd_get_systemunitdir)/systemd-udevd.service.d"

sdk_container/src/third_party/coreos-overlay/coreos/user-patches/sys-apps/systemd/0001-vpick-Don-t-use-openat-directly-but-resolve-symlinks.patch

@@ -0,0 +1,33 @@
+From 6f4b065b626edd8a06ff0c8028173e060b5e444b Mon Sep 17 00:00:00 2001
+From: Kai Lueke <kailuke@microsoft.com>
+Date: Thu, 20 Nov 2025 23:43:55 +0900
+Subject: [PATCH 03/10] vpick: Don't use openat directly but resolve symlinks
+ in given root
+
+With systemd-sysext --root= all symlinks should be followed relative to
+the given root and direct openat usage doesn't work.
+Change the openat call to use the chase helper function to resolve the
+symlink in the given root.
+---
+ src/shared/vpick.c | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/src/shared/vpick.c b/src/shared/vpick.c
+index b1b2d93054..dfe58cafa5 100644
+--- a/src/shared/vpick.c
++++ b/src/shared/vpick.c
+@@ -471,9 +471,9 @@ static int make_choice(
+         if (!p)
+                 return log_oom_debug();
+ 
+-        object_fd = openat(dir_fd, best_filename, O_CLOEXEC|O_PATH);
++        object_fd = chase_and_openat(toplevel_fd, p, CHASE_AT_RESOLVE_IN_ROOT, O_PATH|O_CLOEXEC, NULL);
+         if (object_fd < 0)
+-                return log_debug_errno(errno, "Failed to open '%s/%s': %m",
++                return log_debug_errno(object_fd, "Failed to open '%s/%s': %m",
+                                        empty_to_root(toplevel_path), skip_leading_slash(inode_path));
+ 
+         return pin_choice(
+-- 
+2.52.0
+