Replies: 5 comments 2 replies
-
|
Hello @sidneychang , please take a look of the above plan as discussed earlier and let me know if I missed anything or for any mistakes. |
Beta Was this translation helpful? Give feedback.
-
|
The following content captures my current understanding and findings from this week's exploration of containerd, rootfs setup, and snapshotters. It may still contain incomplete sections or inaccuracies, and I will continue improving and refining it as I explore the topic further. 1. Overall Architecture and Container rootfs Construction Workflow
A typical example appears in // 1. Pull the image and unpack it into the snapshotter
image, err := client.Pull(context, "docker.io/library/redis:latest", containerd.WithPullUnpack)
// 2. Allocate a rootfs and OCI spec for the container based on the image
redis, err := client.NewContainer(context, "redis-master",
containerd.WithNewSnapshot("redis-rootfs", image),
containerd.WithNewSpec(oci.WithImageConfig(image)),
)
// 3. Turn the container object into a runnable task
task, err := redis.NewTask(context, cio.NewCreator(cio.WithStdio))
defer task.Delete(context)
pid := task.Pid()
err = task.Start(context)
status, err := task.Wait(context)This workflow can be abstracted as:
2. rootfs Preparation and the Role of Snapshotter2.1
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you @sidneychang for the detailed explanation. Everything looks good. |
Beta Was this translation helpful? Give feedback.
-
|
Last week I also ran some preliminary startup latency tests in my prototype impletation. In this prototype, the snapshot view is created in the shim before launching urunc. The idea is to prepare a read-only view of the container snapshot and expose the required artifacts (e.g., unikernel, initrd, and configuration files) to the runtime without directly copying them from the snapshot device. To understand the impact of this approach on startup latency, I added instrumentation logs around the snapshot view creation path in the shim. The experiments were conducted using the nginx-qemu-linux-raw image. The results suggest that the dominant cost comes from the devmapper snapshot view creation itself. A typical run shows:
This results in roughly ~170 ms of additional control-plane overhead in the startup path when the snapshot view is created. For comparison, in the previous implementation we directly copied the required files (unikernel, initrd, and This difference is also reflected in the end-to-end startup measurements:
From these preliminary results, the additional latency appears to come mainly from the control-plane operations required to create the snapshot view (interaction with containerd, snapshot service calls, and metadata updates). For relatively small images such as That said, snapshot views may still be beneficial in other scenarios (for example larger images, repeated snapshot reuse, or cases where reducing filesystem duplication is more important than minimizing startup latency). I would be interested to hear whether this observation aligns with the intended usage model of snapshot views, or if there are scenarios where this approach would be expected to perform better. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @sidneychang for this early evaluation. To be honest, I was not expecting the snapshot creation to require that much time. On the other hand, the small overhead of copying files in urunc can be explained from the fact that these operations take place in tmpfs. However, as you already mentioned snapshot views will be beneficial in some scenarios. I do not believe that these scenarios are larger images, since we are only interested in some specific files. However, in case we deploy multiple containers with the same image (and hence reducing duplication), snapshot views will perform better than copying files. For example with the numbers form the early evaluation spawning ~12 containers with the same image could have the same overhead as creating a single snapshot view and sharing it among all of the containers. Also, this overhead will affect entirely the first container and not the rest. Another point that would be helpful to search is if the size of the image affects the time required for the creation of the view snapshot. If that is the case, then we can explore ways to decrease the size of the snapshot we are interested (e.g. only first layer of the image, I am not sure if this is possible). |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the suggestion. I also think that sharing the same snapshot view across multiple containers is a feasible direction. In fact, I had considered this possibility earlier as well. My main concern is how to properly manage the lifecycle of such a view. Since this view would not be created directly by containerd, we would need a mechanism to ensure that containerd can reclaim it at the appropriate time. I will investigate how this could be handled safely. Regarding the snapshot size, I noticed that when containerd creates a view snapshot, most of the work seems to involve database updates and metadata writes. Because of that, I am not entirely sure whether the size of the selected layer actually has a significant impact on the creation time. Even if it does, interacting with containerd through APIs (for example when using lightweight operations like leases) already takes a few milliseconds. Therefore, it might be difficult for snapshot views alone to significantly improve container startup latency. |
Beta Was this translation helpful? Give feedback.
-
|
During some startup latency experiments with urunc + containerd + devmapper snapshotter, I observed several behaviors that might be interesting to share. The goal of the experiment was to explore container startup latency under different snapshot usage patterns. One important note about the environment: the experiments were conducted on a VM with relatively limited memory, but according to system monitoring during the tests, no swap space was used, so the observed latency does not appear to be related to swapping. Testing MethodOriginally, the testing pattern was:
Under this pattern, the startup latency remained relatively stable. Later, to explore snapshot reuse behavior, the testing method was changed to:
Observation 1: Higher Startup Latency When Another Container Is RunningWhen using the devmapper snapshotter, it appears that if there is already a container running based on the same image, starting additional containers tends to have higher startup latency. From instrumentation logs added around the runtime startup path, the increased latency is not due to urunc itself, but mainly comes from Example results (baseline implementation without snapshot views): Compared with the first container startup (~0.8–1.0s), additional containers tend to start around 1.2–1.5s when another container based on the same image is already running. Observation 2: Using a Shared Snapshot ViewI experimented with an approach intended to reuse the snapshot view more explicitly: Create a snapshot view Mount the view block device on the host filesystem Share the mounted filesystem with multiple containers via bind mounts In other words, instead of letting each container mount its own snapshot, multiple containers reuse the same mounted filesystem derived from the snapshot view. In this setup, I observed that container startup latency fluctuates significantly more, again mainly due to TaskService.Create, rather than urunc itself. At this stage, the focus is not on whether the view-based implementation is faster or slower, but rather on the latency variability observed during repeated container startups. Overall, compared with the baseline pattern, the view-based approach showed noticeably larger startup latency variance in the tests. I am continuing to investigate the underlying causes in the containerd snapshotter and runtime startup path, and I will update this thread if I find additional details. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Mentorship plan: Optimize rootfs handling with block-based snapshotters
This discussion outlines the proposed plan as discussed with @sidneychang for the CNCF mentorship project "Optimize rootfs handling with block-based snapshotters" for urunc.
The plan is structured into three phases. Each phase has clear goals and specific outcomes, along with suggested tasks and sub-tasks to help guide the work. The listed sub-tasks are meant as guidance and reference points, they are not strict requirements. The exact order of tasks within each phase can be adjusted as long as the main outcomes are achieved.
Phase 1
The goal of Phase 1 is to build the necessary background and get familiar with containerd and its snapshotters. This phase is expected to last up to 3 weeks (02/03/2026 – 22/03/2026). During this time, the focus will be on understanding contianerd, the various snapshotters, and getting familiar with snapshots.
Tentative tasks and sub-tasks
Outcomes
Deadline: Completed no later than 22/03/2026 (AoE).
Description: A report with:
Phase 2
The goal of Phase 2 is to apply the knowledge gained in Phase 1 to urunc and begin implementing the use of view snapshots for obtaining the required boot files for the monitors. This phase is expected to last up to 5 weeks (23/03/2026 – 03/05/2026). During this time, the focus will shift to the practical implementation within urunc.
Tentative tasks and sub-tasks
Outcomes:
Deadline: Completed no later than 02/05/2026 (AoE).
Description: The following items:
Phase 3
The goal of Phase 3 is to finalize and polish the integration of view snapshots for obtaining the necessary boot files for the monitors.
This phase will also include an evaluation to demonstrate the benefits of the new approach (view snapshots) compared to the previous method (mounting and copying files). This phase is expected to last up to 4 weeks (04/05/2026 – 28/05/2026), with an emphasis on refinement, validation, and broader ecosystem exploration. During this period, the focus will be on refining the implementation, validating the design decisions through evaluation, and exploring related approaches in other high level runtimes.
Tentative tasks and sub-tasks
Outcomes:
Deadline:: Completed no later than 29/05/2026 (AoE).
Description: The following items:
Beta Was this translation helpful? Give feedback.
All reactions