LFx Mentorship (CNCF - urunc: Investigate missing custom OCI annotations in urunc containers)

[cmainas]

Mentorship plan: Investigate missing custom OCI annotations in urunc containers

This discussion outlines the proposed plan as discussed with @ilp-sys for the CNCF mentorship project "Investigate missing custom OCI annotations in urunc containers",

The plan is structured into three phases. Each phase has clear goals and specific outcomes, along with suggested tasks and sub-tasks to help guide the work. The listed sub-tasks are meant as guidance and reference points, they are not strict requirements. The exact order of tasks within each phase can be adjusted as long as the main outcomes are achieved.

Phase 1

The goal of Phase 1 is to build the necessary background and get familiar with OCI annotations, containerd and the annotating container images at build time. This phase is expected to last up to 3 weeks (02/03/2026 – 22/03/2026). During this period, the focus will be on understanding the theoretical aspects of OCI images, establishing the knowledge required to effectively apply these concepts in the later implementation phases.

Tentative tasks and sub-tasks

• Theoretical background (OCI spec):
  • Get familiar with the components of the OCI images spec
  • Identify which parts of the OCI image spec accept annotations and what is their roles
• Get familiar with the annotations in the OCI runtime spec
  • Identify how they are passed to the runtime
  • Identify other mechanisms to store information froom build time to images that are supposed to be visible at runtime
• Hands on building and exploring annotation support from tools:
  • Build custom OCI images with various tools (docker, buildah, containerd, etc.)
  • Find which of the above tools can store annotations and in which parts of the OCI image
  • Check which high level tools do pass the annotations from the OCI images to the OCI runtime (docker, ctr, nerdctl, podman, k8s etc.)
• Identifying the root cause:
  • In the tools and platforms that annotations are not passed correctly, identify why this happens (lack of support, implementation decisions, etc.)
  • Find which of the above tools can store annotations and in which parts of the OCI image

Outcome

Deadline: Completed no later than 22/03/2026 (AoE).
Description: A report with the:

• theoretical background for annotations and all other mechanisms that can be sued
• findings from the hands on exploration with the various tools`
• findings from the search of the root cause

Phase 2

The goal of Phase 2 is to investigate where annotations are getting lost in the containerd / urunc workflow and create a Proof of Concept to resolve the metadata passing from images to the container runtime. This phase is expected to last up to 5 weeks (23/03/2026 – 03/05/2026). During this time, the focus will shift to the practical investigation of the missing OCI annotations and the implementation of a PoC that resolves the overall issue (without or without annotations).

Tentative tasks and sub-tasks

• Getting familiar with urunc and reproducing the issue:
  • Set up an environment with urunc
  • Execute existing images with urunc
  • Build custom images with bunny and execute them with urunc
  • Understand the dependency of annotations in urunc and why urunc requires them
• Under the setup of containerd / urunc:
  • Identify when annotations are visible and when they are not
  • Identify the differences between the each path and what makes annotations invisible
  • Identify where and how containerd stores and retrieves such information (annotations)
• Fixing or overcoming the issue:
  • For containerd identify the high level changes required to make the annotations visible in the low level runtime (if possible)
  • For urunc, find how it can retrieve the information of the annotations from containerd
  • Explore alternative options based on the findings from phase 1
• Create a Proof of Concept
  • Begin the implementation in a branch with the rough changes that resolve the issue.

Outcomes

Deadline: Completed no later than 02/05/2026 (AoE).
Description: The following items:

1. A report with:
   • the findings from the specific exploration in containerd and urunc
   • a plan to resolve the issue, specifying the high level changes required in each tool
2. A PoC that resolves the issue

Phase 3

The goal of Phase 3 is to finalize and polish the solution created in the previous phase. This phase is expected to last up to 4 weeks (04/05/2026 – 28/05/2026), with an emphasis on refinement, validation, and broader ecosystem exploration. During this period, the focus will be on refining the solution created in the previous phase, enhance it with the necessary tests explore ways to overwrite the metadata passed to the runtime at runtime and explore the compatibility of the current approach with other high level container runtimes (e.g. CRI-O),

Tentative tasks and sub-tasks

• Upstream the necessary changes:
  • Form the PoC from rough patches to a series of commits that can be merged.
  • Open PR and iterate over the necessary changes to merge it.
• Enhancing with further tests:
  • Provide the necessary testing for the new changes adding unit and integration tests
  • Create end to end tests to verify the correctness of the solution
• Analyze the side effects:
  • In the case of runtimes analyze how the new changes affect the performance (if applicable)
  • In the case of building tools analyze the difference in the UX (if applicable)
• Expanding the current support:
  • Based on the solution, explore ways to efficiently update the necessary metadata, either at build time or deployment time
  • Implement in the respective tools the changes for such support
  • Implement mechanisms to expand the current form of storing these metadata to support more advanced types of data like arrays.
• Explore compatibility with other high level runtimes
  • Explore if the suggested solution is applicable in other high level container runtimes (e.g. CRI-O)
  • Analyze any further changes required

Outcomes

Deadline: Completed no later than 29/05/2026 (AoE).
Description: The following items:

1. PRs to all the tools with the changes that resolve the issue
2. A report summarizing all the work

[cmainas]

Hello @ilp-sys , please take a look of the above plan as discussed earlier and let me know if I missed anything or for any mistakes.

[ilp-sys]

Sure, thank you for this outline. Let's get started!!

[ilp-sys]

(WIP) Phase 1 Report

• Investigate Missing Custom OCI Annotations in urunc Containers
• Duration) 3/3/26 ~ 10/3/26

Problem Statement

urunc relies on OCI image annotations to pass essential metadata for unikernel execution. While this works in Kubernetes, these annotations fail to propagate in non-K8s environments, forcing the use of alternative workarounds. In this mentorship program, we aim to identify the root cause of this propagation failure and implement a fix to ensure urunc receives the necessary metadata across all container environments.

Theoretical Background

Location of Annotations in OCI Images

To integrate unikernels into container environments, OCI images are used to package both the application and the necessary metadata. The OCI Image Specification
defines that an image is composed of an image index, an image manifest, and an image configuration.

Annotations are optional metadata fields that can appear in three components of the specification, namely in the image index, image manifest and the labels in the image configuration.

OCI image
├─ image index (optional)
│   └─ annotations
├─ image manifest
│   ├─ config (descriptor → image config)
│   ├─ layers
│   └─ annotations
└─ image config (JSON blob)
    └─ config.Labels

The OCI specification also defines that the image configuration object is the primary input used to generate the OCI runtime configuration. As a result, metadata stored in the OCI image does not automatically become visible to the container runtime. Only metadata that is propagated into the OCI runtime specification becomes accessible during container execution.

Annotation Propagation from Image to Runtime

Container engines and runtimes (e.g., Docker, Podman, containerd) are responsible for converting an OCI image into an OCI runtime bundle. During this conversion process, metadata stored in the image may be omitted depending on the implementation.

The general metadata flow can be summarized as follows:

Build-time
   ↓
OCI image
   ├ manifest.annotations
   └ config.labels
   ↓
container engine / runtime
   ↓
OCI runtime bundle
   └ config.json (annotations)
   ↓
low-level runtime (runc / urunc)

Only metadata that is explicitly propagated into the runtime specification (config.json) becomes visible to the low-level container runtime. Metadata stored solely in the image manifest or other image components may not be preserved during this conversion step.

Initial Investigation Results

Initial investigation reports how image labels and OCI annotations are preserved or lost across different container tools and runtime stages. It investigates whether metadata stored in an OCI image is preserved when containers are created and executed using container engines and runtimes such as Docker, Podman, containerd(via ctr and nerdctl), and OCI tooling like umoci.

Previously Reported Observations

Tool / Command	Metadata Level Observed	Labels Present	Annotations Present	Observation
docker buildx imagetools inspect --raw	OCI image manifest (registry)	N/A	Yes	The annotation com.urunc.unikernel.binary is present in the image manifest.
crane manifest	OCI image manifest	N/A	Yes	Confirms that the annotation exists in the registry-stored manifest.
docker manifest inspect	OCI image manifest	N/A	Yes	Same result: annotations are visible at the manifest level.
podman inspect IMAGE	Image metadata	Yes	Yes	Podman exposes both labels and annotations when inspecting the image.
docker inspect IMAGE	Image config	Yes	No	Docker exposes labels from the image config but does not expose manifest annotations.
podman inspect CONTAINER	Container configuration	Yes	Yes	Both labels and annotations remain accessible after container creation.
docker inspect CONTAINER	Container configuration	Yes	No	Labels are preserved but annotations are not propagated.
nerdctl inspect IMAGE	Image metadata	Yes	No	Only labels are visible.
nerdctl inspect CONTAINER	Container metadata	No	No	Neither labels nor annotations are present in container metadata.
ctr c info	containerd container metadata	Yes	No	Labels are stored, but annotations are missing.
containerd runtime config.json	OCI runtime spec	No	No	Runtime configuration does not include labels or annotations from the image.
umoci unpack → bundle/config.json	OCI runtime bundle	Yes	No	Labels appear as annotations in the runtime spec, but manifest annotations are not preserved.
crictl inspecti	CRI image specification	Yes	No	Only image config labels are exposed.

Key findings

1. In Kubernetes-based environments (e.g., Podman workflows), OCI annotations appear to be preserved and accessible at runtime.
2. In containerd-based environments (e.g., Docker, nerdctl, ctr), OCI annotations are not propagated to the container runtime configuration.
3. The behavior observed with umoci unpack also shows that annotations from the image manifest are not propagated into the OCI runtime bundle (config.json), while image labels may still appear.

Further Investigation

In non-Kubernetes deployments, urunc annotations are missing because containerd’s generic image-to-spec path relies on the OCI image configuration and does not propagate manifest-level annotations into spec.Annotations. This is consistent with the OCI Image Specification stating that “a converter SHOULD NOT attempt to extract annotations from manifests or image indices.” The CRI path instead injects Kubernetes annotations as external inputs.

Cases Where Annotations Are Preserved

• cri-path: https://github.com/containerd/containerd/blob/main/internal/cri/server/container_create.go

Cases Where Annotations Are Lost

The loss of annotations can be observed through the following experiment.

First, we verify that the OCI image manifest actually contains annotations.

╰─$ sudo ctr content get sha256:60503bb1db41f94083b99f047448e4b7bcc3ace5c9a4999ec0ef4014ac3c52c9 | jq '.annotations'
{
  "com.test.foo": "bar",
  "org.opencontainers.image.base.digest": "sha256:5c8b2c0a6c745bc177669abfaa716b4bc57d58e2ea3882fb5da67f4d59e3dda5",
  "org.opencontainers.image.base.name": "docker.io/library/ubuntu:22.04"
}

Next, we run a container using the image.

sudo dlv exec ./bin/ctr -- \
  --address /run/containerd/containerd.sock \
  -n default run --rm localhost:5000/test-annotation:latest test1

We can observe that the code responsible for retrieving the image configuration returns only the config descriptor from the manifest and ignores the manifest annotations.

• https://github.com/containerd/containerd/blob/main/core/images/image.go#L262C1-L268C2

• In the following execution path, the image configuration is used to populate the container labels.

   261: // configuration of the image.
   262: func Config(ctx context.Context, provider content.Provider, image ocispec.Descriptor, platform platforms.MatchComparer) (ocispec.Descriptor, error) {
   263:         manifest, err := Manifest(ctx, provider, image, platform)
=> 264:         if err != nil {
   265:                 return ocispec.Descriptor{}, err
   266:         }
   267:         return manifest.Config, nil
   268: }
   269:
(dlv) p manifest
github.com/opencontainers/image-spec/specs-go/v1.Manifest {
        Versioned: github.com/opencontainers/image-spec/specs-go.Versioned {SchemaVersion: 2},                                      
        MediaType: "application/vnd.oci.image.manifest.v1+json",
        ArtifactType: "",
        Config: github.com/opencontainers/image-spec/specs-go/v1.Descriptor {
                MediaType: "application/vnd.oci.image.config.v1+json",
                Digest: "sha256:d47671acdbf35974d8548f4f659ee3a453dfd180b49313d0a51a16410...+7 more",
                Size: 1579,
                URLs: []string len: 0, cap: 0, nil,
                Annotations: map[string]string nil,
                Data: []uint8 len: 0, cap: 0, nil,
                Platform: *github.com/opencontainers/image-spec/specs-go/v1.Platform nil,
                ArtifactType: "",},
        Layers: []github.com/opencontainers/image-spec/specs-go/v1.Descriptor len: 2, cap: 2, [
                (*"github.com/opencontainers/image-spec/specs-go/v1.Descriptor")(0xc00046e0f0),
                (*"github.com/opencontainers/image-spec/specs-go/v1.Descriptor")(0xc00046e168),
        ],
        Subject: *github.com/opencontainers/image-spec/specs-go/v1.Descriptor nil,
        Annotations: map[string]string [
                "com.test.foo": "bar",
                "org.opencontainers.image.base.digest": "sha256:5c8b2c0a6c745bc177669abfaa716b4bc57d58e2ea3882fb5da67f4d5...+7 more",
                "org.opencontainers.image.base.name": "docker.io/library/ubuntu:22.04",
        ],}
(dlv) bt
 0  0x0000000000dc64b3 in github.com/containerd/containerd/v2/core/images.Config
    at ./core/images/image.go:264
 1  0x0000000000dc35be in github.com/containerd/containerd/v2/core/images.(*Image).Config
    at ./core/images/image.go:108
 2  0x0000000000fabdef in github.com/containerd/containerd/v2/client.(*image).Config
    at ./client/image.go:205
 3  0x0000000000fa1b04 in github.com/containerd/containerd/v2/client.WithImageConfigLabels.func1
    at ./client/container_opts.go:118
 4  0x0000000000f8d293 in github.com/containerd/containerd/v2/client.(*Client).NewContainer
    at ./client/client.go:361
 5  0x00000000011a80a6 in github.com/containerd/containerd/v2/cmd/ctr/commands/run.NewContainer
    at ./cmd/ctr/commands/run/run_unix.go:457
 6  0x000000000119f6a5 in github.com/containerd/containerd/v2/cmd/ctr/commands/run.init.func1
    at ./cmd/ctr/commands/run/run.go:184
 7  0x00000000006dd942 in github.com/urfave/cli/v2.(*Command).Run
    at ./vendor/github.com/urfave/cli/v2/command.go:276
 8  0x00000000006dd885 in github.com/urfave/cli/v2.(*Command).Run
    at ./vendor/github.com/urfave/cli/v2/command.go:269
 9  0x00000000006d8a25 in github.com/urfave/cli/v2.(*App).RunContext
    at ./vendor/github.com/urfave/cli/v2/app.go:333
10  0x00000000006d870b in github.com/urfave/cli/v2.(*App).Run
    at ./vendor/github.com/urfave/cli/v2/app.go:307
11  0x000000000134f7e5 in main.main
    at ./cmd/ctr/main.go:32
12  0x00000000004499e7 in runtime.main
    at /home/jiwoo/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/runtime/proc.go:283
13  0x0000000000485601 in runtime.goexit
    at /home/jiwoo/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/runtime/asm_amd64.s:1700

• Another execution paths retrieves the image configuration while setting the OCI runtime specifictaion.

(dlv) n
> github.com/containerd/containerd/v2/core/images.Config() ./core/images/image.go:264 (PC: 0xdc64b3)
   259: //
   260: // The caller can then use the descriptor to resolve and process the
   261: // configuration of the image.
   262: func Config(ctx context.Context, provider content.Provider, image ocispec.Descriptor, platform platforms.MatchComparer) (ocispec.Descriptor, error) {
   263:         manifest, err := Manifest(ctx, provider, image, platform)
=> 264:         if err != nil {
   265:                 return ocispec.Descriptor{}, err
   266:         }
   267:         return manifest.Config, nil
   268: }
   269:
(dlv) p manifest
github.com/opencontainers/image-spec/specs-go/v1.Manifest {
        Versioned: github.com/opencontainers/image-spec/specs-go.Versioned {SchemaVersion: 2},
        MediaType: "application/vnd.oci.image.manifest.v1+json",
        ArtifactType: "",
        Config: github.com/opencontainers/image-spec/specs-go/v1.Descriptor {
                MediaType: "application/vnd.oci.image.config.v1+json",
                Digest: "sha256:d47671acdbf35974d8548f4f659ee3a453dfd180b49313d0a51a16410...+7 more",
                Size: 1579,                                                                                                                               
                URLs: []string len: 0, cap: 0, nil,
                Annotations: map[string]string nil,
                Data: []uint8 len: 0, cap: 0, nil,
                Platform: *github.com/opencontainers/image-spec/specs-go/v1.Platform nil,
                ArtifactType: "",},
        Layers: []github.com/opencontainers/image-spec/specs-go/v1.Descriptor len: 2, cap: 2, [
                (*"github.com/opencontainers/image-spec/specs-go/v1.Descriptor")(0xc000018000),
                (*"github.com/opencontainers/image-spec/specs-go/v1.Descriptor")(0xc000018078),
        ],
        Subject: *github.com/opencontainers/image-spec/specs-go/v1.Descriptor nil,
        Annotations: map[string]string [
                "com.test.foo": "bar",
                "org.opencontainers.image.base.digest": "sha256:5c8b2c0a6c745bc177669abfaa716b4bc57d58e2ea3882fb5da67f4d5...+7 more",
                "org.opencontainers.image.base.name": "docker.io/library/ubuntu:22.04",
        ],}
(dlv) bt
 0  0x0000000000dc64b3 in github.com/containerd/containerd/v2/core/images.Config
    at ./core/images/image.go:264
 1  0x0000000000dc35be in github.com/containerd/containerd/v2/core/images.(*Image).Config
    at ./core/images/image.go:108
 2  0x0000000000fabdef in github.com/containerd/containerd/v2/client.(*image).Config
    at ./client/image.go:205
 3  0x0000000000f0c14b in github.com/containerd/containerd/v2/pkg/oci.WithImageConfigArgs.func1
    at ./pkg/oci/spec_opts.go:372
 4  0x0000000000f06936 in github.com/containerd/containerd/v2/pkg/oci.ApplyOpts
    at ./pkg/oci/spec.go:110
 5  0x0000000000fa3ee5 in github.com/containerd/containerd/v2/client.WithSpec.func1
    at ./client/container_opts.go:318
 6  0x0000000000f8d293 in github.com/containerd/containerd/v2/client.(*Client).NewContainer
    at ./client/client.go:361
 7  0x00000000011a80a6 in github.com/containerd/containerd/v2/cmd/ctr/commands/run.NewContainer
    at ./cmd/ctr/commands/run/run_unix.go:457
 8  0x000000000119f6a5 in github.com/containerd/containerd/v2/cmd/ctr/commands/run.init.func1
    at ./cmd/ctr/commands/run/run.go:184
 9  0x00000000006dd942 in github.com/urfave/cli/v2.(*Command).Run
    at ./vendor/github.com/urfave/cli/v2/command.go:276
10  0x00000000006dd885 in github.com/urfave/cli/v2.(*Command).Run
    at ./vendor/github.com/urfave/cli/v2/command.go:269
11  0x00000000006d8a25 in github.com/urfave/cli/v2.(*App).RunContext
    at ./vendor/github.com/urfave/cli/v2/app.go:333
12  0x00000000006d870b in github.com/urfave/cli/v2.(*App).Run
    at ./vendor/github.com/urfave/cli/v2/app.go:307
13  0x000000000134f7e5 in main.main
    at ./cmd/ctr/main.go:32
14  0x00000000004499e7 in runtime.main
    at /home/jiwoo/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/runtime/proc.go:283
15  0x0000000000485601 in runtime.goexit
    at /home/jiwoo/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/runtime/asm_amd64.s:1700
(dlv)

Root Cause Analysis

Preparation for Phase 2

[ilp-sys]

Hey @cmainas @panosmaurikos ,

I plan to use this discussion to keep track of my progress and findings.

I’ll keep this comment updated as I go through phase 1, so feel free to check in and leave any feedback or thoughts you might have. I want to keep everything transparent and leave a solid record of how we’re approaching this—from the background concepts to the actual implementation.

I’ll reach out again if I run into any questions, thanks!

[cmainas]

Hello @ilp-sys ,

thank you, it sounds like a good idea.

[cmainas]

Hello @ilp-sys,

a small recap of our sync today (10/03/2026):

• The information you found in the OCI spec regarding the propagation of manifest and index annotations is very useful.
• An approach would be to explore if contianerd provides the metadata in config.Label in the OCI runtime config.
• Another (future) approach would be to explore any APIs from containerd that can provide these information.
• It would be helpful to have a function call of containerd and how it handles the annotations in the image manifest (just in a bit more detailed text than the above).

Also, regarding the tracing of Go programs, it might be helpful to check the trace Go package and the respective go tool trace. Using these two you can trace the function calls inside containerd. Moreover, containerd has some documentation about tracing, but I have never tried it out.

At last, here is the
document from Kata-containers that explains how to specify per-Pod configuration for Kata Containers using annotations. However, as we discussed in the call, these annotations are declared at deployment time and hence containerd and other tools properly forward them to Kata Containers.

[ilp-sys]

Hello @cmainas,

I have added the function trace for the non-cri container execution path.
Could you please take a look and let me know if this explanation is clear?
If this looks good, I will write up the cri part in the same manner.

[cmainas]

Hello @ilp-sys ,

thank you for the trace. If I udnerstand correctly the trace you shared refers to the creation of the rootfs for the container. However, we are mostly interested in the trace for the creation of the OCI spec for the runtime. Looking a bit in the code, it should be this variable https://github.com/containerd/containerd/blob/main/cmd/ctr/commands/run/run_unix.go#L124 which is later passed as argument in NewContainer https://github.com/containerd/containerd/blob/main/cmd/ctr/commands/run/run_unix.go#L457. Especially this block looks interesting to investigate more https://github.com/containerd/containerd/blob/main/cmd/ctr/commands/run/run_unix.go#L182-L188

Please let me know if I misunderstood something in the trace.

[ilp-sys]

Hey @cmainas I’ve updated the trace.

Previously I set the breakpoint directly on Config(), which captured the rootfs preparation path. Following your suggestion, the updated trace now shows the OCI runtime spec creation path (WithSpec → ApplyOpts → WithImageConfigArgs). Could you please take another look and let me know if this looks correct?

[cmainas]

Hello @ilp-sys ,

thank you for the update. It seems that the Config function will never return the annotations. However, there is a path in containerd (cri) where the annotations are kept. This is what we are searching for. There is a specific decision which affects if annotations in the image manifest will be stored in the container's config or not. What is different between the cri and non-cri code path in containerd. Also keep in mind that the function call trace is just a way to find faster the differences. If you are more comfortable with code reading, that is also fine. At the end, we just need to have a good understanding of how containerd makes a few decisions about the annotations and if we can replace them with something else.

The questions that we need to answer:

• When does containerd retrieve the image manifest. If annotations are present in the cri path, then at some time containerd does retrieve them. We need to find this point. The Config function you have targeted does not return the annotations, therefore there should be another place.
• Does the point where containerd reads these annotations is present in both cri and non cri execution flow?
• If no where is the decision made and why?
• If yes when does containerd discard the annotations and why?