K8S 1.35: image volume mount subpath ignored caused by containerd < 2.2

[awoimbee]

Image I'm using: Bottlerocket OS 1.54.0 (aws-k8s-1.35)

What I expected to happen:
In K8S 1.35, the feature gate for OCI volumes is enabled by default (release notes).

I tried using this feature:

  volumeMounts:
  - mountPath: /opt/keycloak/themes/mytheme
    name: theme
    subPath: /themes/mytheme
# ...
volumes:
- image:
    reference: myregistry.example.com/keycloak-themes:xyz
  name: theme

What actually happened:
subPath is ignored, the final path looks like /opt/keycloak/themes/mytheme/themes/mytheme/ instead of /opt/keycloak/themes/mytheme.
This is because the subPath feature requires containerd 2.2 and bottlerocket still uses containerd 2.1 (containerd PR introducing the feature).

bash-5.2# containerd --version
containerd github.com/containerd/containerd/v2 2.1.6+bottlerocket c74fd8780002eb26bd5940ae339d690d891221c2

How to reproduce the problem:

Just try to use an OCI volume, subPath is ignored (you can put random values in there it's ignored).

[arnaldo2792]

Hey @awoimbee , thanks for letting us know about this problem! We will track this for when we add containerd 2.2.

[ytsssun]

Still doing more testing for the containerd 2.2 introduction. But I can confirm that in my testing with containerd 2.2 and k8s 1.35, the subPath is being honored.

You can see the test details in the PR bottlerocket-os/bottlerocket-core-kit#862

[runningman84]

It looks like a new version of the core kit was released but it is not included in the ami so far.