Distribute using OCI artifacts / allow shim to pull artifact

[radu-matei]

This is a feature request to allow distribution of apps targeting runwasi using OCI artifacts, and not as a container images.

Currently, all applications targeting runwasi need to be distributed as a container image, with the structure carefully constructed. This is not an ideal long-term solution for a number of reasons, deduplication of Wasm files and static assets being one of them, particularly in the context of bytecodealliance/registry#87 and https://hackmd.io/50rfwV6BTJWN8VZBhdAN_g.

Ideally, pulling the artifact could be done by the actual shim implementation — and implementations could continue to default to wrapping the Wasm app in a container image; however, some shim implementations would benefit greatly from using their existing mechanism of distributing apps using OCI artifacts (see https://developer.fermyon.com/spin/distributing-apps.md#publishing-a-spin-application-to-a-registry).

Thoughts?

cc @rumpl, @squillace, @Mossaka, @devigned.

[jsturtevant]

This has a bit of cross-over of the conversation on deislabs/containerd-wasm-shims#89. In particular, we were looking for ways to enable containerd to do the pulling or unpacking.

There was also @devigned's comment that aligns with your idea that we should be using OCI artifacts and not container images.. That conversation wondered a bit but @cpuguy83 had some thoughts on using the oci bundle.

[squillace]

I absolutely believe this is the proper path. I do wish to support scratch images if possible; but it introduces arch and os issues that have to be navigated and that's precisely "handling the current" and not moving to the proper solution. Artifacts are that solution in my opinion.

[cpuguy83]

To note, there's been some discussion for having containerd ask the shim to pull in order to better support confidential compute.
I think wasm tends to fall in the same category there along with VM's.
It wouldn't be awful to let the shim handle pull so we don't have to teach containerd how to deal with stuff... but then every shim needs to implement that.

[squillace]

Is there a way to surface a shared ability? Some crate?

[radu-matei]

We do have a crate we've been using in both Krustlet and Spin to push and pull Wasm, but it is by no means limited to Wasm — https://github.com/krustlet/oci-distribution. You can see it in use in Spin, and we can easily surface this functionality to the shims — https://github.com/fermyon/spin/tree/main/crates/oci.

[squillace]

I think it would be great to model artifact graphs using this; perhaps a spin as a working example (though also understand a standalone tool) and then runwasi as how that pull would land properly?

[ipuustin]

We do have a crate we've been using in both Krustlet and Spin to push and pull Wasm, but it is by no means limited to Wasm — https://github.com/krustlet/oci-distribution. You can see it in use in Spin, and we can easily surface this functionality to the shims — https://github.com/fermyon/spin/tree/main/crates/oci.

I think containerd registry host definitions are stored in hosts.toml file (see https://github.com/containerd/containerd/blob/main/docs/hosts.md). Probably a crate could parse that file to get pointers to the correct certificates, registry capabilities etc. The file location is part of containerd config and could maybe be passed as an env variable to the shim (there are already several environment variables defined in the the shim runtime spec).

Also containerd gets the HTTP(s) proxy settings from environment variables, so the shims might already inherit them. We just need to make sure that the no_proxy values are parsed right and that we use the same variable name scheme as containerd.

[cpuguy83]

containerd (starting at 1.7) has an API called the "transfer service" which is pretty flexible in terms of customizing its behavior from the client.
We already have the containerd socket passed down to the shim so we should, in theory, be able to use that without weird issues.
If it is missing functionality that we need we should definitely work upstream to resolve that.

[jsturtevant]

I worked on a few proto-types with the various ideas that have been proposed across the different issues and came up with the following (thanks @cpuguy83 for the help understanding all he different ways to approach this in containerd): https://docs.google.com/document/d/11shgC3l6gplBjWF1VJCWvN_9do51otscAm0hBDGSSAc. It has some prototype code and a demo linked in the document.

I've added it to the agenda for our next runwasi meeting on May 23rd. Please take a look and give feedback 👍