From 7288dd562ed622d7521f01f262d103a767b57919 Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Thu, 31 Mar 2022 08:53:27 -0500 Subject: [PATCH 001/222] Stop Deleting Cache Images Signed-off-by: Jesse Brown --- text/0000-stop-deleting-cache-images.md | 73 +++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 text/0000-stop-deleting-cache-images.md diff --git a/text/0000-stop-deleting-cache-images.md b/text/0000-stop-deleting-cache-images.md new file mode 100644 index 000000000..edd35bd70 --- /dev/null +++ b/text/0000-stop-deleting-cache-images.md @@ -0,0 +1,73 @@ +# Meta +[meta]: #meta +- Name: Stop deleting cache images +- Start Date: 2022-03-31 +- Author(s): jabrown85 +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: [buildpacks/lifecycle#803](https://github.com/buildpacks/lifecycle/issues/803) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +Lifecycle will no longer try to delete previous cache images when publishing cache images. + +# Definitions +[definitions]: #definitions + +* Cache Image: The cache stored between builds - stored in a registry. +* ECR: Amazon's container registry product + +# Motivation +[motivation]: #motivation + +- Why should we do this? + As discussed in [buildpacks/lifecycle#803](https://github.com/buildpacks/lifecycle/issues/803), some registries (ECR) do not support `DELETE`. For platforms that work exclusively with such registries, the warning output by lifecycle and the time taken to fail is unavoidable and lifecycle is wasting time trying to complete an operation that will never succeed. + +- What use cases does it support? + All platforms that use cache images against registries that do not support delete will no longer see warning messages. + +- What is the expected outcome? + Platforms will need to handle cleanup of their cache images on their own, if they desire. + +# What it is +[what-it-is]: #what-it-is + +Lifecycle will no longer attempt to delete cache images during cache image export. + +# How it Works +[how-it-works]: #how-it-works + +Lifecycle will no longer attempt to delete cache images during cache image export. + +# Migration +[migration]: #migration + +Lifecycle will document this behavior change in Release Notes/Changelog along with the associated Platform API that enables the new behavior. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +Platform authors relying on this behavior will need to take additional measures to ensure cache image cleanup. + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + * Add regex or configuration to drive registry hosts to ignore during DELETE * +- Why is this proposal the best? + * Lifecycle is not currently cleaning up any other resources + * Deleting the cache images can hurt reproducibility + * There are more public registries that don't allow DELETE +- What is the impact of not doing this? + * End users continue seeing warnings the platform can do nothing about. + + +# Prior Art +[prior-art]: #prior-art + +Discussion at [buildpacks/lifecycle#803](https://github.com/buildpacks/lifecycle/issues/803). From e58a18334470fcb9780e91f3e068d73a34de5775 Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Thu, 31 Mar 2022 08:53:27 -0500 Subject: [PATCH 002/222] fixup! Stop Deleting Cache Images Signed-off-by: Jesse Brown --- text/0000-stop-deleting-cache-images.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-stop-deleting-cache-images.md b/text/0000-stop-deleting-cache-images.md index edd35bd70..0fe8f5653 100644 --- a/text/0000-stop-deleting-cache-images.md +++ b/text/0000-stop-deleting-cache-images.md @@ -52,7 +52,7 @@ Lifecycle will document this behavior change in Release Notes/Changelog along wi Why should we *not* do this? -Platform authors relying on this behavior will need to take additional measures to ensure cache image cleanup. +Platform authors relying on this behavior will need to take additional measures to ensure cache image cleanup or the destination registry will continue to grow. # Alternatives [alternatives]: #alternatives From 4badd26916dd9a1dfa78f0d3cdd3d3fc6a51ea21 Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Thu, 31 Mar 2022 08:53:27 -0500 Subject: [PATCH 003/222] added alternative Signed-off-by: Jesse Brown --- text/0000-stop-deleting-cache-images.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-stop-deleting-cache-images.md b/text/0000-stop-deleting-cache-images.md index 0fe8f5653..b90027bfe 100644 --- a/text/0000-stop-deleting-cache-images.md +++ b/text/0000-stop-deleting-cache-images.md @@ -59,6 +59,7 @@ Platform authors relying on this behavior will need to take additional measures - What other designs have been considered? * Add regex or configuration to drive registry hosts to ignore during DELETE * + * Stop deleting cache images by default in newer platform API versions, but add a platform-level configuration to enable previous behavior. - Why is this proposal the best? * Lifecycle is not currently cleaning up any other resources * Deleting the cache images can hurt reproducibility From 04db1526da5aa0dc03c467e8444de957ba1ab400 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 21 Jun 2022 11:53:28 -0500 Subject: [PATCH 004/222] Initial version of the proposal to donate kpack to CNB project Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 313 +++++++++++++++++++++++++++++ 1 file changed, 313 insertions(+) create mode 100644 text/0000-kpack-donation-to-cnb.md diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md new file mode 100644 index 000000000..2fd41ec7f --- /dev/null +++ b/text/0000-kpack-donation-to-cnb.md @@ -0,0 +1,313 @@ +# Meta + +- Name: kpack donation to CNB +- Start Date: 2022-06-21 +- Author(s): [Juan Bustamante](https://github.com/jjbustamante/) +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary + +This RFC proposes the donation of the open-source project [kpack](https://github.com/pivotal/kpack/) into the [Cloud Native Buildpack](https://buildpacks.io/) (CNB) ecosystem. + +# Definitions + +- [Kubernetes](https://kubernetes.io/) is an open-source system for automating deployment, scaling, and management of containerized applications. +- [Kpack](https://github.com/pivotal/kpack/) is a VMware-led open-source project that utilizes [Kubernetes](https://kubernetes.io/) primitives to build OCI images as a [platform](https://buildpacks.io/docs/concepts/components/platform/) implementation of [Cloud Native Buildpacks](https://buildpacks.io/). +- A Kubernetes native application is an application designed to run on Kubernetes platforms, managed by Kubernetes APIs and `kubectl` tooling and cohesively deployed on Kubernetes as a single object. + +# Motivation + +### Why should we do this? + +It will benefit the [CNB](https://buildpacks.io/) project by adding a tool to support an out-of-the box [Kubernetes](https://kubernetes.io/) integration, which is part of the [CNB](https://buildpacks.io/) [roadmap](https://github.com/buildpacks/community/blob/main/ROADMAP.md#integration-with-the-cloud-native-ecosystem) goals. + +It will show evidence to the community that the project supports multiple [platform interface specification](https://github.com/buildpacks/spec/blob/main/platform.md) implementers increasing community's confidence on the flexibility of specification maintained by the [CNB](https://buildpacks.io/) project. + +It will help the [CNB](https://buildpacks.io/) community (+550 members on slack channel) to grow by adding all the [kpack](https://github.com/pivotal/kpack/) community into [CNB](https://buildpacks.io/) space. + +[CNB](https://buildpacks.io/) is part of the [Cloud Native Computing Foundation](https://www.cncf.io), an open source, vendor neutral hub of cloud native computing projects, the inclusion of [kpack](https://github.com/pivotal/kpack/) under this umbrella will provide more opportunity to the community: + +- Increase in adopters, users looking to use buildpacks in [Kubernetes](https://kubernetes.io/) will find a tool supported and maintained by the [CNB team](https://github.com/buildpacks/community/blob/main/TEAMS.md). +- Improve efficiency, ensuring that the roadmaps of the two projects are closer aligned will make it easier to coordinate efforts between both communities. + +### What use cases does it support? + +[kpack](https://github.com/pivotal/kpack/) will add support to operators by providing declarative [Kubernetes](https://kubernetes.io/) resources (images, builders, or stacks for example) to monitor for security patches on the underlying builder's buildpacks or stacks and rebuild the OCI image when changes are detected, allowing platforms to roll out new versions of the applications when vulnerabilities are fixed. + +### How does kpack support the goals and use cases of the project? + +The [CNB](https://buildpacks.io/) project turns application source code into OCI-compliant container images; in order to do that, it defines a platform-to-buildpack contract that guarantees interoperability between different implementers. + +The [CNB](https://buildpacks.io/) project embraces modern container standards, and [Kubernetes](https://kubernetes.io/) has become the industry standard for automating deployment, scaling, and management of containerized applications. + +[kpack](https://github.com/pivotal/kpack/) fits perfectly in that direction because it implements the [platform interface specification](https://github.com/buildpacks/spec/blob/main/platform.md) and because is a [Kubernetes](https://kubernetes.io/) native application its community possesses a vast knowledge that can provide valuable feedback to the CNB project. + +### Is there functionality in kpack that is already provided by the project? + +[pack](https://github.com/buildpacks/pack) and [kpack](https://github.com/pivotal/kpack/) offer similar functionality (both tools implement the [platform interface](https://github.com/buildpacks/spec/blob/main/platform.md)[specification](https://github.com/buildpacks/spec/blob/main/platform.md)) but they do it for two non-overlapping contexts: while the first one targets developers and local builds, [kpack](https://github.com/pivotal/kpack/) manages containerization on day-2 and at scale and is a [Kubernetes](https://kubernetes.io/) native implementation. + +### Is kpack integrated with another service or technology that is widely used? + +As mentioned earlier, [kpack](https://github.com/pivotal/kpack/) implements the [platform interface specification](https://github.com/buildpacks/spec/blob/main/platform.md) on [Kubernetes](https://kubernetes.io/), a standard nowadays for automating deployment, scaling, and management of containerized applications. + +# What it is + +[Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/) is a portable, extensible, open-source platform for managing containerized workloads and services. The [Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/) API can be extended in different ways; one of them is using [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/), a custom resource represents a customization of a particular [Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/) installation. + +[kpack](https://github.com/pivotal/kpack/) extends [Kubernetes](https://kubernetes.io/) using [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) and utilizes unprivileged [Kubernetes](https://kubernetes.io/) primitives to provide builds of OCI images as a platform implementation of [Cloud Native Buildpacks](https://buildpacks.io/). This means that [kpack](https://github.com/pivotal/kpack/) takes the CNB-defined concepts (image, builder, stacks, etc) and bakes them into the Kubernetes extension model using custom resources and exposing a declarative API for interacting with it. + +The declarative API enforces a separation of responsibilities. Operators declare the configuration for a CNB image or define which buildpacks or stacks must be used, and [kpack](https://github.com/pivotal/kpack/) - using its custom controller - will take care of the heavy lifting, keeping the state of the custom objects in sync with the declared desired state. + +# How it Works + +As mentioned before, [kpack](https://github.com/pivotal/kpack/) uses the [custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) extension point to provide the capabilities of building OCI images as a platform implementation of [Cloud Native Buildpacks](https://buildpacks.io/). + +These custom resources have a common definition similar to this: + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: [ClusterStack|ClusterStore|Image|Builder|Build] +metadata: + name: [unique name] +``` + +The _apiVersion_ key specifies which version of the Kubernetes API is used to create the object, in this case **kpack.io/v1alpha2** + +The _kind_ key specifies what kind of objects we want to create for example: **ClusterStack, ClusterStore, Image, Builder or Build** + +The _metadata_ key is used to define the data that can uniquely identify the object. One common key used around all the custom resources is to provide a _name_ to identify the object. + +Some of the [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) implemented by [kpack](https://github.com/pivotal/kpack/) are describe in the next section, if you want to check the complete reference go to [kpack](https://github.com/pivotal/kpack/) documentation [site](https://github.com/pivotal/kpack/tree/main/docs) + +## ClusterStack + +This resource is an abstraction to group a `build image` and a `run image` required to build the application source code. + +Let's see an example of the [ClusterStack](https://github.com/pivotal/kpack/blob/main/docs/stack.md) definition + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: ClusterStack +metadata: + name: base +spec: + id: "io.buildpacks.stacks.bionic" + buildImage: + image: "my-buildpack-repo/build:cnb" + runImage: + image: "my-buildpack-repo/run:cnb" +``` + +The _spec_ key is used to define the desired state of the ClusterStack and the keys availables under _spec_ match the values expected in a CNB [stack](https://buildpacks.io/docs/concepts/components/stack/) definition: + +- _id_: The 'id' of the stack +- _buildImage.image_: The `build-image` of the [stack](https://buildpacks.io/docs/concepts/components/stack/). +- _runImage.image_: The `run-image` of the [stack](https://buildpacks.io/docs/concepts/components/stack/). + +## Cluster Store + +Creates a repository of buildpacks packaged as OCI artifacts to be used during a build. + +Let's see an example of the [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) definition + +``` yaml +apiVersion: kpack.io/v1alpha2 +kind: ClusterStore +metadata: + name: my-cluster-store +spec: + sources: + - image: foo.com/my-buildpack-repo/buildpack-1@sha256:sha123 + - image: foo.com/my-buildpack-repo/buildpack-2@sha256:sha345 + - image: foo.com/my-buildpack-repo/builder:base + ``` + +The _spec_ key is used to define the desired state of the [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) + +- _sources_: List of buildpackage images to make available in the ClusterStore. Each image is an object with the key _image_. + +As a side note the [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) resource will be deprecated in favor of a new Buildpack resource in the near future according to the following [RFC](https://www.google.com/url?q=https://github.com/pivotal/kpack/pull/931&sa=D&source=docs&ust=1665521917723122&usg=AOvVaw1eNN-XzLf5xiX1nvrHKMRE) + +## Builder or ClusterBuilder + +Creates a [CNB builder](https://buildpacks.io/docs/concepts/components/builder/) image that contains all the components necessary to execute a build. + +An example of the [Builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) definition is as follows: + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: Builder +metadata: + name: my-builder +spec: + tag: foo.com/sample/builder + stack: + name: base + kind: ClusterStack + store: + name: my-cluster-store + kind: ClusterStore + order: + - group: + - id: my-buildpack-repo/buildpack-1 + - group: + - id: my-buildpack-repo/buildpack-2 + ``` + +It's important to notice that a [ClusterStack](https://github.com/pivotal/kpack/blob/main/docs/stack.md) and [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) is required for creating a [Builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md). + +The _spec_ key is used to define the desired state of the [Builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) + +- _tag_: The tag to save the builder image. +- _stack.name_: The name of the stack resource to use as the builder stack. All buildpacks in the order must be compatible with the [ClusterStack](https://github.com/pivotal/kpack/blob/main/docs/stack.md). +- _stack.kind_: The type as defined in [Kubernetes](https://kubernetes.io/). This will always be [ClusterStack](https://github.com/pivotal/kpack/blob/main/docs/stack.md). +- _store.name_: The name of the [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md) resource in [Kubernetes](https://kubernetes.io/). +- _store.kind_: The type as defined in [Kubernetes](https://kubernetes.io/). This will always be [ClusterStore](https://github.com/pivotal/kpack/blob/main/docs/store.md). +- _order_: The [builder order](https://buildpacks.io/docs/reference/builder-config/). + +The [ClusterBuilder](https://github.com/pivotal/kpack/blob/main/docs/builders.md#cluster-builders) resource is almost identical to a [Builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) but it is a cluster scoped resource that can be referenced by an [Image](https://github.com/pivotal/kpack/blob/main/docs/image.md) in any namespace. + +## Build + +Custom resource responsible for scheduling and running a single build. + +An example of a [Build](https://github.com/pivotal/kpack/blob/main/docs/build.md) definition is + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: Build +metadata: + name: sample-build +spec: + tags: + -sample/image + builder: + image: foo.com/sample/builder + projectDescriptorPath: path/to/project.toml + source: + git: + url: https://github.com/my-account/sample-app.git + revision: main +``` + +The _spec_ key is used to define the desired state of the [Build](https://github.com/pivotal/kpack/blob/main/docs/build.md) + +- _tags_: A list of tags to build. At least one tag is required. +- _builder.image_: This is the tag to the [Cloud Native Buildpacks builder image](https://buildpacks.io/docs/concepts/components/builder/) to use in the build. +- _source_: The source location that will be the input to the build. +- _projectDescriptorPath_: Path to the [project descriptor file](https://buildpacks.io/docs/reference/config/project-descriptor/) relative to source root dir or subPath if set. + +## Image + +Provides a configuration to build and maintain an OCI image utilizing [CNB](https://buildpacks.io/). + +An example of an [Image](https://github.com/pivotal/kpack/blob/main/docs/image.md) definition is as follows + +```yaml +apiVersion: kpack.io/v1alpha2 +kind: Image +metadata: + name: my-app-image + namespace: default +spec: + tag: foo.com/my-app-repo/my-app-image + builder: + name: my-builder + kind: Builder + source: + git: + url: https://github.com/my-account/sample-app.git + revision: 82cb521d636b282340378d80a6307a08e3d4a4c4 +``` + +The _spec_ key is used to define the desired state of the [Image](https://github.com/pivotal/kpack/blob/main/docs/image.md) + +- _tag_: The image tag. +- _builder_: Configuration of the [builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) resource the image builds will use. +- source: The source code that will be monitored/built into images. + +# Migration + +### Repositories + +The suggested strategy for migrating [kpack's](https://github.com/pivotal/kpack/) git repositories to the [CNB](https://buildpacks.io/) is to use the [transfer repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-organization) git feature. + +The following table shows the candidates repositories to be transferred + +| Origin Repo | Description | Owner | Destination Repo | Owner | +| --- | --- | --- | --- | --- | +| [https://github.com/pivotal/kpack](https://github.com/pivotal/kpack) | kpack source code | Pivotal | [https://github.com/buildpacks/kpack](https://github.com/buildpacks/kpack) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | +| [https://github.com/vmware-tanzu/kpack-cli](https://github.com/vmware-tanzu/kpack-cli) | kpack CLI | VMware | [https://github.com/buildpacks/kpack-cli](https://github.com/buildpacks/kpack-cli) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | +| [https://github.com/vmware-tanzu/homebrew-kpack-cli](https://github.com/vmware-tanzu/homebrew-kpack-cli) | Homebrew tap for the kpack CLI | VMware | [https://github.com/buildpacks/homebrew-kpack-cli](https://github.com/buildpacks/homebrew-kpack-cli) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | + +For each repository + +- The owner or admin user must follow the steps describe in github [documentation](https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-organization) and transfer the repository to the organization [Cloud Native Buildpacks](https://github.com/buildpacks) +- A member of the [TOC team](https://github.com/orgs/buildpacks/teams/toc/members) in [CNB](https://buildpacks.io/) must accept the donation of the repository. The name of the destination repository will be the one described in the table above. + +### CI / CD Pipelines + +[kpack's](https://github.com/pivotal/kpack/) CI/CD infrastructure currently runs on internal VMware infrastructure. [kpack's](https://github.com/pivotal/kpack/) CI/CD pipelines will need to be rebuilt on [CNB](https://buildpacks.io/) infrastructure. + +### Documentation + +[Kpack](https://github.com/pivotal/kpack/) documentation is currently hosted in the base code [repository](https://github.com/pivotal/kpack/tree/main/docs), after migrating to [CNB](https://buildpacks.io/) the documentation will be published into the Cloud Native Buildpack [site](https://buildpacks.io/). + +[CNB](https://buildpacks.io/) already mentioned [kpack](https://github.com/pivotal/kpack/) in their documentation, specifically, in the tools section. The proposal is: + +- Create a new folder name **kpack** inside the [tool](https://github.com/buildpacks/docs/tree/main/content/docs/tools) section in the docs repository +- Copy kpack's [documentation](https://github.com/pivotal/kpack/tree/main/docs) into this new created folder +- Update the references and all the required elements to format the documentation according to [CNB](https://buildpacks.io/) site + +### Governance + +Based on the [CNB governance policy](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md) and the fact that [kpack](https://github.com/pivotal/kpack/) is a [platform](https://buildpacks.io/docs/concepts/components/platform/) implementation of [Cloud Native Buildpacks](https://buildpacks.io/), it will be added under the responsibility of the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team). + +How do migrate roles and responsibilities into the CNB governance process? + +Currently the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team) already has a **team lead** assigned and, by definition, each team can have only one **team lead**. In order to provide the current [kpack](https://github.com/pivotal/kpack/) team with the same accountability for the migrated repositories the proposal is to follow the guidelines describe on the [Component Maintainer Role RFC](https://github.com/buildpacks/rfcs/pull/234) + +# Risks + +- It's not clear how to handle the budget required to finance the infrastructure to rebuild the CI/CD pipelines on CNCF CNB infrastructure. +- Evaluate any legal requirement from [CNCF](https://www.cncf.io) that must be fulfilled before accepting the project into the [CNB](https://buildpacks.io/) ecosystem. +- Relying on the approval of [Component Maintainer Role RFC](https://github.com/buildpacks/rfcs/pull/234) to guarantee current [kpack](https://github.com/pivotal/kpack/) maintainers could keep supporting the project after the donation. + +# Drawbacks + +Why should we _not_ do this? + +- If the [CNB](https://buildpacks.io/) team expects to implement a different kind of integration with [Kubernetes](https://kubernetes.io/), then accepting the donation of [kpack](https://github.com/pivotal/kpack/) could conflict with that strategy. +- Another component to maintain which requires additional context and expertise in [Kubernetes](https://kubernetes.io/). + +# Alternatives + +- What other designs have been considered? + - [VMware](https://www.vmware.com/) could continue to control the project, but it doesn't help on increase the adoption because it remains as a single-vendor driven project + - [VMware](https://www.vmware.com/) could donate [kpack](https://github.com/pivotal/kpack/) to the [Continuous Delivery Foundation](https://cd.foundation/), but [CNB](https://buildpacks.io/) presents a natural home for [kpack](https://github.com/pivotal/kpack/) (it is an implementation of the platform specification) + - [VMware](https://www.vmware.com/) could create a new [CNCF](https://www.cncf.io/) project and move all [kpack](https://github.com/pivotal/kpack/) resources to it, but in this case it would need to undergo as a sandbox project for example. + +- Why is this proposal the best? + +[kpack](https://github.com/pivotal/kpack/) is a mature Kubernetes-native tool that leverages buildpacks and is used in production environments. The project's maintainers and contributors possess valuable technical and user context, derived from developing [kpack](https://github.com/pivotal/kpack/) and integrating feedback from users utilizing [CNB](https://buildpacks.io/) concepts when presented as part of Kubernetes resources. + +- What is the impact of not doing this? + +The [CNB](https://buildpacks.io/) community would have to develop from scratch any kind of integration with the Cloud Native Ecosystem to satisfy the project goals. + +**Prior Art** + +- Guidelines for accepting component-level contributions [RFC #143](https://github.com/buildpacks/rfcs/pull/143) +- Component Maintainer Role [RFC #234](https://github.com/buildpacks/rfcs/pull/234) + +# Unresolved Questions + +See the risks section + +# Spec. Changes (OPTIONAL) + +None From a8aed9eec457ecde45ee73e254ca660cad831bf0 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 18 Oct 2022 18:15:10 -0500 Subject: [PATCH 005/222] WIP - adding slack channel and RFC sections Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 48 ++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 2fd41ec7f..f62cabc16 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -265,12 +265,59 @@ For each repository ### Governance +#### Team roles + Based on the [CNB governance policy](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md) and the fact that [kpack](https://github.com/pivotal/kpack/) is a [platform](https://buildpacks.io/docs/concepts/components/platform/) implementation of [Cloud Native Buildpacks](https://buildpacks.io/), it will be added under the responsibility of the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team). How do migrate roles and responsibilities into the CNB governance process? Currently the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team) already has a **team lead** assigned and, by definition, each team can have only one **team lead**. In order to provide the current [kpack](https://github.com/pivotal/kpack/) team with the same accountability for the migrated repositories the proposal is to follow the guidelines describe on the [Component Maintainer Role RFC](https://github.com/buildpacks/rfcs/pull/234) +#### RFC process + +Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will follow the [RFC process](https://github.com/buildpacks/rfcs) stablished in CNB project for any new RFC required by the project. + +##### Template + +[kpack](https://github.com/pivotal/kpack/) have their own RFC process, let's compare both templates + +| [kpack template](https://github.com/pivotal/kpack/blob/main/rfcs/0000-template.md) | [CNB template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) mapping | +| ----------------------- | ------------------------------ | +| Problem | Motivation | +| Outcome | Motivation | +| Actions to take | What it is / How it works | +| Complexity | - | +| - | Migration | +| Prior Art | Prior Art | +| Alternatives | Alternatives | +| Risks | Drawbacks | +| - | Unresolved Questions | +| - | Spec. Changes | +| - | History| + +As we can see the [CNB RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) consider most of the sections required by the [kpack RFC](https://github.com/pivotal/kpack/blob/main/rfcs/0000-template.md) process, the only one missing is: + - **complexity**: What thoughts do you have on the complexity of this? + +The proposal is to incorporate this section into the [CNB template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md). + +##### Existing RFC + +- Open: +- Closed: + +#### Slack channel + +The proposal is to move the [kpack](https://github.com/pivotal/kpack/) slack instance from the [Kubernetes slack instance](https://kubernetes.slack.com/channels/kpack) to the [CNCF slack instance](https://slack.cncf.io/). + +[kpack](https://github.com/pivotal/kpack/) maintainers will have to request the migration of the [kpack](https://github.com/pivotal/kpack/) slack over to [CNCF slack](https://slack.cncf.io/) and co-ordinate the announcements/user migration. + +The following are the slack channels that will be migrated over and their proposed names after migrations + +| Current | Proposed | +| ----------------------- | ------------------------------ | +| kpack | buildpacks-kpack | + + # Risks - It's not clear how to handle the budget required to finance the infrastructure to rebuild the CI/CD pipelines on CNCF CNB infrastructure. @@ -303,6 +350,7 @@ The [CNB](https://buildpacks.io/) community would have to develop from scratch a - Guidelines for accepting component-level contributions [RFC #143](https://github.com/buildpacks/rfcs/pull/143) - Component Maintainer Role [RFC #234](https://github.com/buildpacks/rfcs/pull/234) +- Proposal to move CNCF slack [RFC #198](https://github.com/buildpacks/rfcs/pull/198) # Unresolved Questions From 71efdee835f654a964f4c1f7b7b836078c6357b5 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 19 Oct 2022 09:51:06 -0500 Subject: [PATCH 006/222] adding RFC process proposal Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 31 +++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index f62cabc16..f9581e38b 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -253,6 +253,13 @@ For each repository [kpack's](https://github.com/pivotal/kpack/) CI/CD infrastructure currently runs on internal VMware infrastructure. [kpack's](https://github.com/pivotal/kpack/) CI/CD pipelines will need to be rebuilt on [CNB](https://buildpacks.io/) infrastructure. +##### Hardware requirements + +The minimal hardware requirements to request to CNCF to recreate the CI/CD pipelines are: + +- TBD + + ### Documentation [Kpack](https://github.com/pivotal/kpack/) documentation is currently hosted in the base code [repository](https://github.com/pivotal/kpack/tree/main/docs), after migrating to [CNB](https://buildpacks.io/) the documentation will be published into the Cloud Native Buildpack [site](https://buildpacks.io/). @@ -275,13 +282,24 @@ Currently the [CNB Platform Team](https://github.com/buildpacks/community/blob/m #### RFC process -Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will follow the [RFC process](https://github.com/buildpacks/rfcs) stablished in CNB project for any new RFC required by the project. +Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will follow the [RFC process](https://github.com/buildpacks/rfcs) stablished in CNB project for any new RFC created in the project. + +##### Existing RFC + +- Open: Currently there are < 10 open RFCs (some of them opened 2 years ago). + - The proposal is to suggest the [kpack](https://github.com/pivotal/kpack/) maintainers to: + - Triage those RFCs an update their status before the donation. + - Co-ordinate the announcement of the donation to the RFCs authors and explain them the strategy after the migration (next section) + - After the donation, any open RFCs in [kpack](https://github.com/pivotal/kpack/) repository should be closed + - The RFC author should create a new RFC in the CNB RFC [repository](https://github.com/buildpacks/rfcs) and follow the CNB [RFC process](https://github.com/buildpacks/rfcs) + +- Closed: For historical purpose, we will keep those RFC in the repo ##### Template -[kpack](https://github.com/pivotal/kpack/) have their own RFC process, let's compare both templates +The following table shows the mapping between the sections of the current kpack [RFC template](https://github.com/pivotal/kpack/blob/main/rfcs/0000-template.md) vs the CNB [RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) -| [kpack template](https://github.com/pivotal/kpack/blob/main/rfcs/0000-template.md) | [CNB template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) mapping | +| kpack template | CNB template | | ----------------------- | ------------------------------ | | Problem | Motivation | | Outcome | Motivation | @@ -295,16 +313,11 @@ Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will | - | Spec. Changes | | - | History| -As we can see the [CNB RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) consider most of the sections required by the [kpack RFC](https://github.com/pivotal/kpack/blob/main/rfcs/0000-template.md) process, the only one missing is: +As we can see the CNB [RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) consider most of the sections required by the kpack [RFC](https://github.com/pivotal/kpack/blob/main/rfcs/0000-template.md) process, the only missing one is: - **complexity**: What thoughts do you have on the complexity of this? The proposal is to incorporate this section into the [CNB template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md). -##### Existing RFC - -- Open: -- Closed: - #### Slack channel The proposal is to move the [kpack](https://github.com/pivotal/kpack/) slack instance from the [Kubernetes slack instance](https://kubernetes.slack.com/channels/kpack) to the [CNCF slack instance](https://slack.cncf.io/). From 99041240b0ef9bac2c73b809958c8d6da5fc1ffd Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 19 Oct 2022 09:54:34 -0500 Subject: [PATCH 007/222] adding RFC process proposal Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index f9581e38b..fb63c677f 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -286,14 +286,14 @@ Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will ##### Existing RFC -- Open: Currently there are < 10 open RFCs (some of them opened 2 years ago). +- **Open**: Currently there are less that 10 [open RFCs](https://github.com/pivotal/kpack/pulls?q=is%3Apr+label%3ARFC+is%3Aopen) (some of them opened 2 years ago) in [kpack](https://github.com/pivotal/kpack/) repository. - The proposal is to suggest the [kpack](https://github.com/pivotal/kpack/) maintainers to: - Triage those RFCs an update their status before the donation. - Co-ordinate the announcement of the donation to the RFCs authors and explain them the strategy after the migration (next section) - After the donation, any open RFCs in [kpack](https://github.com/pivotal/kpack/) repository should be closed - The RFC author should create a new RFC in the CNB RFC [repository](https://github.com/buildpacks/rfcs) and follow the CNB [RFC process](https://github.com/buildpacks/rfcs) -- Closed: For historical purpose, we will keep those RFC in the repo +- **Closed**: For historical purpose, we will keep those RFC in the repository. ##### Template From 96202b1a84c22f56e64e08ba8c1319e09d368d40 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 21 Oct 2022 11:22:20 -0500 Subject: [PATCH 008/222] Removing the template RFC section after Matthew comments Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 25 +------------------------ 1 file changed, 1 insertion(+), 24 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index fb63c677f..2fc098305 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -282,7 +282,7 @@ Currently the [CNB Platform Team](https://github.com/buildpacks/community/blob/m #### RFC process -Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will follow the [RFC process](https://github.com/buildpacks/rfcs) stablished in CNB project for any new RFC created in the project. +Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will follow the [RFC process](https://github.com/buildpacks/rfcs) and [RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) stablished in CNB project for any new RFC created in the project. ##### Existing RFC @@ -295,29 +295,6 @@ Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will - **Closed**: For historical purpose, we will keep those RFC in the repository. -##### Template - -The following table shows the mapping between the sections of the current kpack [RFC template](https://github.com/pivotal/kpack/blob/main/rfcs/0000-template.md) vs the CNB [RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) - -| kpack template | CNB template | -| ----------------------- | ------------------------------ | -| Problem | Motivation | -| Outcome | Motivation | -| Actions to take | What it is / How it works | -| Complexity | - | -| - | Migration | -| Prior Art | Prior Art | -| Alternatives | Alternatives | -| Risks | Drawbacks | -| - | Unresolved Questions | -| - | Spec. Changes | -| - | History| - -As we can see the CNB [RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) consider most of the sections required by the kpack [RFC](https://github.com/pivotal/kpack/blob/main/rfcs/0000-template.md) process, the only missing one is: - - **complexity**: What thoughts do you have on the complexity of this? - -The proposal is to incorporate this section into the [CNB template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md). - #### Slack channel The proposal is to move the [kpack](https://github.com/pivotal/kpack/) slack instance from the [Kubernetes slack instance](https://kubernetes.slack.com/channels/kpack) to the [CNCF slack instance](https://slack.cncf.io/). From 2e33f4f5f39863019ae93a3d4e22eeb728a3180a Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 21 Oct 2022 12:31:54 -0500 Subject: [PATCH 009/222] adding feedback into the slack channel section Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 2fc098305..e513ed39d 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -297,16 +297,13 @@ Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will #### Slack channel -The proposal is to move the [kpack](https://github.com/pivotal/kpack/) slack instance from the [Kubernetes slack instance](https://kubernetes.slack.com/channels/kpack) to the [CNCF slack instance](https://slack.cncf.io/). +The proposals are: + - `keep` the [kpack](https://github.com/pivotal/kpack/) slack instance from the [Kubernetes slack instance](https://kubernetes.slack.com/channels/kpack), as [kpack](https://github.com/pivotal/kpack/) is a Kubernetes native application most of their users already use [Kubernetes slack instance](https://kubernetes.slack.com/channels/kpack) for communication. + - `create` a new channel in the [CNCF slack instance](https://slack.cncf.io/), this will bring the two communities (kpack and CNB) together -[kpack](https://github.com/pivotal/kpack/) maintainers will have to request the migration of the [kpack](https://github.com/pivotal/kpack/) slack over to [CNCF slack](https://slack.cncf.io/) and co-ordinate the announcements/user migration. - -The following are the slack channels that will be migrated over and their proposed names after migrations - -| Current | Proposed | -| ----------------------- | ------------------------------ | -| kpack | buildpacks-kpack | +[kpack](https://github.com/pivotal/kpack/) maintainers should include the notification of the new channel in the announcement of the donation. +[Platform maintainers](https://github.com/buildpacks/community/blob/main/TEAMS.md#maintainers-1) will have to request or create the new slack channel with the following name: **buildpacks-kpack** (which will be defined as the preferred channel to be used). # Risks From 9f7ac8e50836bc0c59025ec17bbfe1bb47819f90 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 25 Oct 2022 10:33:33 -0500 Subject: [PATCH 010/222] adding hardware requirements Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index e513ed39d..b88fb3a12 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -257,8 +257,9 @@ For each repository The minimal hardware requirements to request to CNCF to recreate the CI/CD pipelines are: -- TBD - +###### Kubernetes cluster + - 3 nodes: 2 CPU / 8 GB RAM / 100 GB ephemeral storage per node + - At least 100 GB of storage in a public OCI registry ### Documentation From a696346207cc803f489ad4f0e7ec09cbfffaf46f Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 26 Oct 2022 15:56:14 -0500 Subject: [PATCH 011/222] updating hardware requirements for CI/CD Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index b88fb3a12..e840c81ad 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -257,9 +257,15 @@ For each repository The minimal hardware requirements to request to CNCF to recreate the CI/CD pipelines are: -###### Kubernetes cluster - - 3 nodes: 2 CPU / 8 GB RAM / 100 GB ephemeral storage per node - - At least 100 GB of storage in a public OCI registry +###### Kubernetes clusters + +**Build cluster** + +- Linux nodes + - 1 amd64 node / 2 CPU / 8GB memory / 50GB ephemeral disk storage +- Windows nodes + - 1 amd64 node / 4 CPU / 16GB memory / 100GB ephemeral disk storage +- At least 100 GB of storage in a public OCI registry ### Documentation From d25a8e9a2a2f5a5d42a24a31181ffabf1a8b98be Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 9 Nov 2022 14:51:17 -0500 Subject: [PATCH 012/222] adding current kpack maintainers table Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index e840c81ad..b405e8053 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -287,6 +287,14 @@ How do migrate roles and responsibilities into the CNB governance process? Currently the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team) already has a **team lead** assigned and, by definition, each team can have only one **team lead**. In order to provide the current [kpack](https://github.com/pivotal/kpack/) team with the same accountability for the migrated repositories the proposal is to follow the guidelines describe on the [Component Maintainer Role RFC](https://github.com/buildpacks/rfcs/pull/234) +The [kpack's](https://github.com/pivotal/kpack/) maintainers that will be nominated as **component maintainer** in CNB are: + +| Name | Github account | Organization | +| --- | --- | --- | +| Matthew McNew| [@matthewmcnew](https://github.com/matthewmcnew)| VMware| +| Tyler Phelan | [@tylerphelan](https://github.com/tylerphelan)| VMware| +| Tom Kennedy | [@tomkennedy513](https://github.com/tomkennedy513) | VMware | + #### RFC process Once the migration is completed, [kpack](https://github.com/pivotal/kpack/) will follow the [RFC process](https://github.com/buildpacks/rfcs) and [RFC template](https://github.com/buildpacks/rfcs/blob/main/0000-template.md) stablished in CNB project for any new RFC created in the project. @@ -316,7 +324,7 @@ The proposals are: - It's not clear how to handle the budget required to finance the infrastructure to rebuild the CI/CD pipelines on CNCF CNB infrastructure. - Evaluate any legal requirement from [CNCF](https://www.cncf.io) that must be fulfilled before accepting the project into the [CNB](https://buildpacks.io/) ecosystem. -- Relying on the approval of [Component Maintainer Role RFC](https://github.com/buildpacks/rfcs/pull/234) to guarantee current [kpack](https://github.com/pivotal/kpack/) maintainers could keep supporting the project after the donation. + # Drawbacks From eb565bb16251ac4a2cd90211206a2a04fb7fcd25 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 9 Nov 2022 17:34:28 -0500 Subject: [PATCH 013/222] Adding some feedback from Joe Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index b405e8053..fa905b2ac 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -230,6 +230,13 @@ The _spec_ key is used to define the desired state of the [Image](https://github - _builder_: Configuration of the [builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) resource the image builds will use. - source: The source code that will be monitored/built into images. +# Adopters + +[VMware](https://www.vmware.com/) is the most obvious company using [kpack](https://github.com/pivotal/kpack/), but there are also other companies like: + +- [Bloomberg](http://techatbloomberg.com/opensource) +- [WPengine](https://wpengine.com/) + # Migration ### Repositories @@ -322,10 +329,10 @@ The proposals are: # Risks +- So far the main company behind [kpack](https://github.com/pivotal/kpack/) is [VMware](https://www.vmware.com/), a reduction in the investment from [VMware](https://www.vmware.com/) would create a problem and the CNB project would have to either sunset [kpack](https://github.com/pivotal/kpack/) or find investment from the community. - It's not clear how to handle the budget required to finance the infrastructure to rebuild the CI/CD pipelines on CNCF CNB infrastructure. - Evaluate any legal requirement from [CNCF](https://www.cncf.io) that must be fulfilled before accepting the project into the [CNB](https://buildpacks.io/) ecosystem. - # Drawbacks Why should we _not_ do this? From 18b926cb784a57c13c5b157023a745e1b173f768 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 30 Nov 2022 14:06:43 -0500 Subject: [PATCH 014/222] adding feedback from Joe Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index fa905b2ac..1f8d82462 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -230,12 +230,27 @@ The _spec_ key is used to define the desired state of the [Image](https://github - _builder_: Configuration of the [builder](https://github.com/pivotal/kpack/blob/main/docs/builders.md) resource the image builds will use. - source: The source code that will be monitored/built into images. +# Contributors + +Contributions to [kpack](https://github.com/pivotal/kpack/) during the period 2022-2019 can be summarized as follow + +```mermaid +pie showData + title Pull Requests Open or Closed + "VMWare or Pivotal" : 438 + "Others" : 37 +``` + # Adopters -[VMware](https://www.vmware.com/) is the most obvious company using [kpack](https://github.com/pivotal/kpack/), but there are also other companies like: +[VMware](https://www.vmware.com/) is the most obvious company using [kpack](https://github.com/pivotal/kpack/) as a core component in their [Tanzu](https://tanzu.vmware.com/tanzu) offer. [kpack](https://github.com/pivotal/kpack/) has been used in the enterprise for years in hundreds of costumers. + +Some others companies using [kpack](https://github.com/pivotal/kpack/) are: - [Bloomberg](http://techatbloomberg.com/opensource) - [WPengine](https://wpengine.com/) +- [Terasky](https://www.terasky.com/) +- [SAP](https://www.sap.com/) # Migration @@ -299,8 +314,19 @@ The [kpack's](https://github.com/pivotal/kpack/) maintainers that will be nomina | Name | Github account | Organization | | --- | --- | --- | | Matthew McNew| [@matthewmcnew](https://github.com/matthewmcnew)| VMware| -| Tyler Phelan | [@tylerphelan](https://github.com/tylerphelan)| VMware| | Tom Kennedy | [@tomkennedy513](https://github.com/tomkennedy513) | VMware | +| Yael Harel | [@yaelharel](https://github.com/yaelharel) | VMware| +| Daniel Chen |[@chenbh](https://github.com/chenbh) | VMware| +| Juan Bustamante |[@jjbustamante](https://github.com/jjbustamante) | VMware| + +Also those members are willing to become more involved with CNB projects and become **Platform maintainers** in the near future. + +Outside VMware, the following contributors manifested their desired to become [kpack's](https://github.com/pivotal/kpack/) **component maintainer**. + +| Name | Github account | Organization | +| --- | --- | --- | +| Sambhav Kothari| [@samj1912](https://github.com/samj1912) | Bloomberg | +| Aidan Delaney| [@AidanDelaney](https://github.com/AidanDelaney) | Bloomberg | #### RFC process From b8330bf54676e7f2b3a30e50235530a571b89aad Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 8 Dec 2022 15:49:30 -0600 Subject: [PATCH 015/222] allow rebase by image digest Signed-off-by: Joey Brown --- text/0000-rebase-immutable-image-ref.md | 110 ++++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 text/0000-rebase-immutable-image-ref.md diff --git a/text/0000-rebase-immutable-image-ref.md b/text/0000-rebase-immutable-image-ref.md new file mode 100644 index 000000000..11070a509 --- /dev/null +++ b/text/0000-rebase-immutable-image-ref.md @@ -0,0 +1,110 @@ +# Meta +[meta]: #meta +- Name: Rebase by Image Digest Reference +- Start Date: 2022-12-08 +- Author(s): [@joeybrown-sf](https://github.com/joeybrown-sf) +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: N/A + +# Summary +[summary]: #summary + +This is a feature for the implementation team. + +Allow passing a digest reference as rebase target. + +# Definitions +[definitions]: #definitions + +An **image reference** refers to either a **tag reference** or **digest reference**. + +A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. + +A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. + + +# Motivation +[motivation]: #motivation + +- This feature will support a workflow where the rebase executing agent only has the digest available to it. + +# What it is +[what-it-is]: #what-it-is + +This provides a high level overview of the feature. + +- Define any new terminology. +- Define the target persona: buildpack author, buildpack user, platform operator, platform implementor, and/or project contributor. +- Explaining the feature largely in terms of examples. +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. +- If applicable, describe the differences between teaching this to existing users and new users. + +# How it Works +[how-it-works]: #how-it-works + +First, let's get some context. Today, we can execute rebase by using **tag references** but not **digest references**. + +Here are some examples of valid rebase commands using **tag references**: +1. `lifecycle rebase my-repo/foo` +1. `lifecycle rebase my-repo/foo:latest` +1. `lifecycle rebase my-repo/foo:v4` + +Here are some examples of currently invalid rebase commands using **digest references**: +1. `lifecycle rebase my-repo/foo@sha256:1234 myrepo/foo` +1. `lifecycle rebase my-repo/foo@sha256:1234 myrepo/foo:latest` +1. `lifecycle rebase my-repo/foo@sha256:1234 myrepo/foo:vNext` + +Supporting Rebase by Image Digest Reference will make both sets of commands valid. + +- When using a digest reference as the image target, there must be at least one tag reference to apply to exported image. +- **If** the `rebase target` is a `digest reference` and one or more `` inputs are provided, each `` MUST refer to the same repository as the `rebase target`. +- If a `` is included without a ``, latest will be used. + +# Migration +[migration]: #migration + +This is backwards compatible. + +# Drawbacks +[drawbacks]: #drawbacks + +# Alternatives +[alternatives]: #alternatives + +# Prior Art +[prior-art]: #prior-art + +`pack` explicitly does not support this. There is validation in `pack`: +` is not a tag reference` + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + + +# History +[history]: #history + + \ No newline at end of file From b29020c19591f4eb0de0a327498c3b6c6e62b104 Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 8 Dec 2022 16:13:19 -0600 Subject: [PATCH 016/222] filled out missing section Signed-off-by: Joey Brown --- text/0000-rebase-immutable-image-ref.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/text/0000-rebase-immutable-image-ref.md b/text/0000-rebase-immutable-image-ref.md index 11070a509..3318c0bf7 100644 --- a/text/0000-rebase-immutable-image-ref.md +++ b/text/0000-rebase-immutable-image-ref.md @@ -34,18 +34,17 @@ A **digest reference** refers to a [content addressable](https://en.wikipedia.o # What it is [what-it-is]: #what-it-is -This provides a high level overview of the feature. +This is a feature to expand the lifecycle rebase command to allow targeting an image by either `tag` or `digest`. -- Define any new terminology. -- Define the target persona: buildpack author, buildpack user, platform operator, platform implementor, and/or project contributor. -- Explaining the feature largely in terms of examples. -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. -- If applicable, describe the differences between teaching this to existing users and new users. +Today, we get the following error when using a digest reference: +``` +ERROR: failed to rebase: failed to write image to the following tags: [localhost:5003/foo/bar@sha256:916a9e100569ee521b86d03b8499b9b93d7d256d6e838868ae720295f2ea2f76: PUT http://localhost:5003/v2/foo/bar/manifests/sha256:916a9e100569ee521b86d03b8499b9b93d7d256d6e838868ae720295f2ea2f76: DIGEST_INVALID: provided digest did not match uploaded content] +``` # How it Works [how-it-works]: #how-it-works -First, let's get some context. Today, we can execute rebase by using **tag references** but not **digest references**. +Today, we can execute rebase by using **tag references** but not **digest references**. Here are some examples of valid rebase commands using **tag references**: 1. `lifecycle rebase my-repo/foo` @@ -60,8 +59,7 @@ Here are some examples of currently invalid rebase commands using **digest refer Supporting Rebase by Image Digest Reference will make both sets of commands valid. - When using a digest reference as the image target, there must be at least one tag reference to apply to exported image. -- **If** the `rebase target` is a `digest reference` and one or more `` inputs are provided, each `` MUST refer to the same repository as the `rebase target`. -- If a `` is included without a ``, latest will be used. +- If a `` is included without a ``, the tag `latest` will be used. # Migration [migration]: #migration From 3200668fe8fdb7bf7f84229d2f55d94d165d3f25 Mon Sep 17 00:00:00 2001 From: Aidan Delaney Date: Mon, 19 Dec 2022 06:31:43 +0000 Subject: [PATCH 017/222] Fix delimiter in RFC 109 The delimiter used in an example is a colon not a double quote character. Signed-off-by: Aidan Delaney --- text/0109-build-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0109-build-config.md b/text/0109-build-config.md index d531e447e..6cf4290c4 100644 --- a/text/0109-build-config.md +++ b/text/0109-build-config.md @@ -58,7 +58,7 @@ Final value: `FOO=test` Buildpack value: `FOO=test` -Build config: `FOO.append=another-value, FOO.delim="` +Build config: `FOO.append=another-value, FOO.delim=:` Final value: `FOO=test:another-value` Buildpack value: `FOO=test` From 740522202a4c239910607f8c3524b1eb0b7c5424 Mon Sep 17 00:00:00 2001 From: Aidan Delaney Date: Mon, 19 Dec 2022 07:40:32 +0000 Subject: [PATCH 018/222] Rename Build Config env variable Use the name `CNB_BUILD_CONFIG_DIR` as provided in the sample implementation. Signed-off-by: Aidan Delaney --- text/0109-build-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0109-build-config.md b/text/0109-build-config.md index 6cf4290c4..0a6ebd0b2 100644 --- a/text/0109-build-config.md +++ b/text/0109-build-config.md @@ -118,5 +118,5 @@ N/A Addition of the definition of the above directory in the Platform specification i.e. - -- `CNB_CONFIG_DIR` +- `CNB_BUILD_CONFIG_DIR` - `/cnb/config/env.build` From 6f4a4a1ea6badbb83212b821ffc6703cd0b4f757 Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Fri, 16 Dec 2022 09:52:56 -0600 Subject: [PATCH 019/222] pass an explicit argument of `-previous-image` Signed-off-by: Joey Brown --- text/0000-rebase-immutable-image-ref.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/text/0000-rebase-immutable-image-ref.md b/text/0000-rebase-immutable-image-ref.md index 3318c0bf7..a8874f080 100644 --- a/text/0000-rebase-immutable-image-ref.md +++ b/text/0000-rebase-immutable-image-ref.md @@ -51,15 +51,15 @@ Here are some examples of valid rebase commands using **tag references**: 1. `lifecycle rebase my-repo/foo:latest` 1. `lifecycle rebase my-repo/foo:v4` -Here are some examples of currently invalid rebase commands using **digest references**: -1. `lifecycle rebase my-repo/foo@sha256:1234 myrepo/foo` -1. `lifecycle rebase my-repo/foo@sha256:1234 myrepo/foo:latest` -1. `lifecycle rebase my-repo/foo@sha256:1234 myrepo/foo:vNext` +It is not currently possible to target an image using **digest references**. -Supporting Rebase by Image Digest Reference will make both sets of commands valid. +Supporting Rebase by Image Digest Reference will provide a mechanism to target an image rebase by tag reference or digest reference. -- When using a digest reference as the image target, there must be at least one tag reference to apply to exported image. -- If a `` is included without a ``, the tag `latest` will be used. +Here is what targeting an image via digest will look like: +1. `lifecycle rebase -previous-image my-repo/foo@sha256:1234 -tag my-repo/foo:rebase my-repo/foo` + +- When using a digest reference as the image target, there may be one or more `` to apply to exported image. If no `tag` is provided, `latest` will be used. +- If `-previous-image` is not provided, the previous is infered from the first argument, just like `analyzer`, for instance. # Migration [migration]: #migration From 1cb7ee0e397348fc66966f24d06a5657cb1e49be Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Wed, 4 Jan 2023 11:47:30 -0600 Subject: [PATCH 020/222] updated some wording Signed-off-by: Joey Brown --- text/0000-rebase-immutable-image-ref.md | 26 +++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/text/0000-rebase-immutable-image-ref.md b/text/0000-rebase-immutable-image-ref.md index a8874f080..3173ba06b 100644 --- a/text/0000-rebase-immutable-image-ref.md +++ b/text/0000-rebase-immutable-image-ref.md @@ -46,20 +46,29 @@ ERROR: failed to rebase: failed to write image to the following tags: [localhost Today, we can execute rebase by using **tag references** but not **digest references**. -Here are some examples of valid rebase commands using **tag references**: -1. `lifecycle rebase my-repo/foo` -1. `lifecycle rebase my-repo/foo:latest` -1. `lifecycle rebase my-repo/foo:v4` +Here are some examples of valid rebase commands. **Tag** is `latest` if not specified: -It is not currently possible to target an image using **digest references**. +``` +lifecycle rebase my-repo/foo +``` +``` +lifecycle rebase my-repo/foo:latest +``` +``` +lifecycle rebase my-repo/foo:v4 +``` + +It is not currently possible to target an image using a **digest reference**. Supporting Rebase by Image Digest Reference will provide a mechanism to target an image rebase by tag reference or digest reference. Here is what targeting an image via digest will look like: -1. `lifecycle rebase -previous-image my-repo/foo@sha256:1234 -tag my-repo/foo:rebase my-repo/foo` +``` +lifecycle rebase -previous-image my-repo/foo@sha256:1234 -tag my-repo/foo:rebase my-repo/foo +``` - When using a digest reference as the image target, there may be one or more `` to apply to exported image. If no `tag` is provided, `latest` will be used. -- If `-previous-image` is not provided, the previous is infered from the first argument, just like `analyzer`, for instance. +- If `-previous-image` is not provided, it is infered from the first argument. This is similar behavior to `analyzer`, for instance. # Migration [migration]: #migration @@ -75,7 +84,8 @@ This is backwards compatible. # Prior Art [prior-art]: #prior-art -`pack` explicitly does not support this. There is validation in `pack`: +`pack` explicitly does not support this. There is a friendly validation message in `pack`: + ` is not a tag reference` # Unresolved Questions From 0447b0b0e5ef542678e6f117f4629aa275a568ff Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Wed, 4 Jan 2023 12:01:48 -0600 Subject: [PATCH 021/222] clarified some verbiage Signed-off-by: Joey Brown --- text/0000-rebase-immutable-image-ref.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/text/0000-rebase-immutable-image-ref.md b/text/0000-rebase-immutable-image-ref.md index 3173ba06b..a21679a2c 100644 --- a/text/0000-rebase-immutable-image-ref.md +++ b/text/0000-rebase-immutable-image-ref.md @@ -12,8 +12,6 @@ # Summary [summary]: #summary -This is a feature for the implementation team. - Allow passing a digest reference as rebase target. # Definitions @@ -29,18 +27,20 @@ A **digest reference** refers to a [content addressable](https://en.wikipedia.o # Motivation [motivation]: #motivation -- This feature will support a workflow where the rebase executing agent only has the digest available to it. +Enables rebasing by targeting an immutable image digest. There are some scenarios where the **digest reference** is preferred over **tag reference**. # What it is [what-it-is]: #what-it-is This is a feature to expand the lifecycle rebase command to allow targeting an image by either `tag` or `digest`. -Today, we get the following error when using a digest reference: +Today, `lifecycle` returns the following error when appempting to use a **digest reference**: ``` ERROR: failed to rebase: failed to write image to the following tags: [localhost:5003/foo/bar@sha256:916a9e100569ee521b86d03b8499b9b93d7d256d6e838868ae720295f2ea2f76: PUT http://localhost:5003/v2/foo/bar/manifests/sha256:916a9e100569ee521b86d03b8499b9b93d7d256d6e838868ae720295f2ea2f76: DIGEST_INVALID: provided digest did not match uploaded content] ``` +This error could be avoided if digest references were permitted. + # How it Works [how-it-works]: #how-it-works @@ -60,14 +60,14 @@ lifecycle rebase my-repo/foo:v4 It is not currently possible to target an image using a **digest reference**. -Supporting Rebase by Image Digest Reference will provide a mechanism to target an image rebase by tag reference or digest reference. +_The proposed feature will provide a mechanism to target an image rebase by tag reference or digest reference._ Here is what targeting an image via digest will look like: ``` lifecycle rebase -previous-image my-repo/foo@sha256:1234 -tag my-repo/foo:rebase my-repo/foo ``` -- When using a digest reference as the image target, there may be one or more `` to apply to exported image. If no `tag` is provided, `latest` will be used. +- When using a digest reference as the image target, the caller may specify zero or more `` to apply to exported image. If no `tag` is provided, `latest` will be used. - If `-previous-image` is not provided, it is infered from the first argument. This is similar behavior to `analyzer`, for instance. # Migration From 6bf522eeee1706ef1e7b341726745fa19ddd27fb Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Thu, 5 Jan 2023 11:23:25 -0600 Subject: [PATCH 022/222] RFC 0115 [#262] Signed-off-by: Jesse Brown --- ...able-image-ref.md => 0115-rebase-immutable-image-ref.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-rebase-immutable-image-ref.md => 0115-rebase-immutable-image-ref.md} (95%) diff --git a/text/0000-rebase-immutable-image-ref.md b/text/0115-rebase-immutable-image-ref.md similarity index 95% rename from text/0000-rebase-immutable-image-ref.md rename to text/0115-rebase-immutable-image-ref.md index a21679a2c..02d548cb7 100644 --- a/text/0000-rebase-immutable-image-ref.md +++ b/text/0115-rebase-immutable-image-ref.md @@ -3,10 +3,10 @@ - Name: Rebase by Image Digest Reference - Start Date: 2022-12-08 - Author(s): [@joeybrown-sf](https://github.com/joeybrown-sf) -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#262](https://github.com/buildpacks/rfcs/pull/262) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: [buildpacks/lifecycle#983](https://github.com/buildpacks/lifecycle/issues/983) - Supersedes: N/A # Summary From cd0920666082c5da5edb43e3d99dd1f3de6f16a0 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 5 Jan 2023 15:44:40 -0500 Subject: [PATCH 023/222] RFC 0116 [#216] Signed-off-by: Natalie Arellano --- ...g-cache-images.md => 0116-stop-deleting-cache-images.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-stop-deleting-cache-images.md => 0116-stop-deleting-cache-images.md} (92%) diff --git a/text/0000-stop-deleting-cache-images.md b/text/0116-stop-deleting-cache-images.md similarity index 92% rename from text/0000-stop-deleting-cache-images.md rename to text/0116-stop-deleting-cache-images.md index b90027bfe..821ca38df 100644 --- a/text/0000-stop-deleting-cache-images.md +++ b/text/0116-stop-deleting-cache-images.md @@ -3,10 +3,10 @@ - Name: Stop deleting cache images - Start Date: 2022-03-31 - Author(s): jabrown85 -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#216](https://github.com/buildpacks/rfcs/pull/216) - CNB Pull Request: (leave blank) -- CNB Issue: [buildpacks/lifecycle#803](https://github.com/buildpacks/lifecycle/issues/803) +- CNB Issue: N/A - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Summary From 9a5fb56d0558d87b9cdaf324b23b45bffc81700d Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 5 Jan 2023 16:37:32 -0500 Subject: [PATCH 024/222] Specify that creating a tracking issue is a step in the merge process Signed-off-by: Natalie Arellano --- README.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 5692aae70..919f58676 100644 --- a/README.md +++ b/README.md @@ -45,12 +45,13 @@ Once an RFC has been accepted, the sub-team maintainers should: Once an `issues-created/` label has been created for each sub-team, the RFC is ready to merge. The team member who merges the pull request should do the following: 1. Assign an id based off the pull request number. -1. Rename the file based off the ID inside `text/`. -1. Fill in the remaining metadata at the top. -1. Commit everything. -1. Update issues with RFC ID and a link to the text file. -1. Update any links in PR description to point at the committed file. -1. Remove the "Final Comment Period" label. +2. Rename the file based off the ID inside `text/`. +3. Fill in the remaining metadata at the top. +4. Commit everything. +5. Update issues with RFC ID and a link to the text file. +6. Update any links in PR description to point at the committed file. +7. Remove the "status/voting" label. +8. Create a [tracking issue](https://github.com/buildpacks/rfcs/issues/new?assignees=&labels=type%2Ftracking&template=tracking.md&title=%5BRFC+%23%3CINSERT+RFC+NUMBER+-+e.g.%2C+0099%3E%5D+-+%3CINSERT+RFC+TITLE%3E). ## Automation @@ -60,4 +61,6 @@ The `merge-rfc.sh` script automates several steps of the merge process for accep ``` Each `` should be of the form `/#` (e.g. `buildpacks/spec#1`). In the rare case that no work must be done in the project as a result of the RFC pass the `-n` flag to explicitly indicate that no issues should be linked. -After running the `merge-rfc.sh` script, manually verify the output before pushing changes. +After running the `merge-rfc.sh` script: +* Manually verify the output before pushing changes. +* Create a [tracking issue](https://github.com/buildpacks/rfcs/issues/new?assignees=&labels=type%2Ftracking&template=tracking.md&title=%5BRFC+%23%3CINSERT+RFC+NUMBER+-+e.g.%2C+0099%3E%5D+-+%3CINSERT+RFC+TITLE%3E). From b55afc0d4be7a5a1d1803090a3b3951e3a400224 Mon Sep 17 00:00:00 2001 From: Aidan Delaney Date: Fri, 6 Jan 2023 09:05:06 +0000 Subject: [PATCH 025/222] Update directory in Build Config RFC We are triangulating this RFC, the implementation and the platform specification. This change makes the RFC agree with the implementation. Signed-off-by: Aidan Delaney --- text/0109-build-config.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0109-build-config.md b/text/0109-build-config.md index 0a6ebd0b2..ebb6537b5 100644 --- a/text/0109-build-config.md +++ b/text/0109-build-config.md @@ -12,7 +12,7 @@ # Summary [summary]: #summary -This RFC proposes an easy way to configure build images to allow specifying a `/cnb/config/env.build` CNB environment directory that allows updating the Buildpack `detect` and `build` environment based on the directory. +This RFC proposes an easy way to configure build images to allow specifying a `/cnb/build-config` CNB environment directory that allows updating the Buildpack `detect` and `build` environment based on the directory. # Definitions @@ -40,10 +40,10 @@ The environment variables may ideally also take precendence over any user provid # What it is [what-it-is]: #what-it-is -The RFC proposes the introduction of the following directory `/cnb/config/env.build` in build images. The directory follows the same convention as a `CNB environment directory`. The notable difference is that the environment variables sourced from this directory are applied **AFTER** processing the user-provided platform environment variables i.e. they should have the highest precedence. These variables should be available during both `detect` and `build` phases (and the `generate` phase in the future). +The RFC proposes the introduction of the following directory `/cnb/build-config/env.build` in build images. The directory follows the same convention as a `CNB environment directory`. The notable difference is that the environment variables sourced from this directory are applied **AFTER** processing the user-provided platform environment variables i.e. they should have the highest precedence. These variables should be available during both `detect` and `build` phases (and the `generate` phase in the future). -The operator can define this directory in the build image under `/cnb/config` or `CNB_CONFIG_DIR` if defined. +The operator can define this directory in the build image under `/cnb/build-config` or `CNB_BUILD_CONFIG_DIR` if defined. # How it Works [how-it-works]: #how-it-works @@ -119,4 +119,4 @@ N/A Addition of the definition of the above directory in the Platform specification i.e. - - `CNB_BUILD_CONFIG_DIR` -- `/cnb/config/env.build` +- `/cnb/build-config/` From 530ba8cda4b5a917a099180276589f46fc2c0a0d Mon Sep 17 00:00:00 2001 From: Aidan Delaney Date: Fri, 6 Jan 2023 17:59:34 +0000 Subject: [PATCH 026/222] Update text/0109-build-config.md Co-authored-by: Natalie Arellano Signed-off-by: Aidan Delaney --- text/0109-build-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0109-build-config.md b/text/0109-build-config.md index ebb6537b5..fde9e5e8f 100644 --- a/text/0109-build-config.md +++ b/text/0109-build-config.md @@ -40,7 +40,7 @@ The environment variables may ideally also take precendence over any user provid # What it is [what-it-is]: #what-it-is -The RFC proposes the introduction of the following directory `/cnb/build-config/env.build` in build images. The directory follows the same convention as a `CNB environment directory`. The notable difference is that the environment variables sourced from this directory are applied **AFTER** processing the user-provided platform environment variables i.e. they should have the highest precedence. These variables should be available during both `detect` and `build` phases (and the `generate` phase in the future). +The RFC proposes the introduction of the following directory `/cnb/build-config/env` in build images. The directory follows the same convention as a `CNB environment directory`. The notable difference is that the environment variables sourced from this directory are applied **AFTER** processing the user-provided platform environment variables i.e. they should have the highest precedence. These variables should be available during both `detect` and `build` phases (and the `generate` phase in the future). The operator can define this directory in the build image under `/cnb/build-config` or `CNB_BUILD_CONFIG_DIR` if defined. From 97cb85517f7283beb92bf505696eebfa76000704 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 25 Jan 2023 17:40:39 -0600 Subject: [PATCH 027/222] initial 2023H1 Roadmap RFC Signed-off-by: Terence Lee --- text/0000-2023H1-roadmap.md | 162 ++++++++++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 text/0000-2023H1-roadmap.md diff --git a/text/0000-2023H1-roadmap.md b/text/0000-2023H1-roadmap.md new file mode 100644 index 000000000..0e0684e4f --- /dev/null +++ b/text/0000-2023H1-roadmap.md @@ -0,0 +1,162 @@ +# Meta +[meta]: #meta +- Name: 2023H1 Roadmap +- Start Date: 2023-01-24 +- Author(s): hone +- Status: Draft Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +This RFC details the first half of the 2023 Roadmap as well as changing the roadmaps to twice a year aligning with KubeCon EU and NA (CDD - Conference Driven Development). + +# Definitions +[definitions]: #definitions + +N/A + +# Motivation +[motivation]: #motivation + +The project has done annual roadmaps in the past, but they haven't been without their challenges. The items range from concrete to large themes. The larger scoped items like "Integration with the Cloud Native Ecosystem" is nebulous and not always clear what success look at the end of it. The roadmap has felt like a smaller vision document versus a set of prioritized items to be accomplished. + +Without clear guidance on how to shape the roadmap, it's also been a challenge to get one out (not that we're doing great this year). We missed 2022's roadmap altogether and didn't publish one until March for 2021. + +Once we did publish a roadmap, items wouldn't necessarily move forward even though we commited as a project to work on them. Items didn't always have an owner and there was no skin in the game for suggesting an idea. + +In addition, we don't do a good job of reviewing our roadmap whether that's regularly through the year or at the end of the year. + +As an incubation level project, there is an opportunity to broadcast announcements at both KubeCon EU and NA. As a project, we're often caught flatfooted with sharing highlights or announcements during these conferences. + + +# What it is +[what-it-is]: #what-it-is + +## Roadmap Changes + +For 2023, I want to propose trying something different this time around. The project will publish two roadmaps one for each half of the year aligned with work to be completed for KubeCon EU (2023-05-16) and KubeCon NA (2023-10-24). The goals I'm hoping to see: + +- Focus - Decrease the amount we're doing as a project, so we're able to deliver on the things we commit to. +- Accountability - With that focus, we also need to be accountable for what we're putting on this list. +- Marketing - Everyone involved works hard to make this project successful. We should take time to celebrate and talk about the work being done. + +### Smaller Scope + +Going forward, the roadmap will account for 6 months (in this case it's just under 4 months) worth of work. This forces items on the roadmap to be smaller and concrete since they have to be something that can be accomplished in that time frame. While it was nice having the large items, they were hard to execute and it wasn't clear what the finished state looked like. Any larger piece work will need to broken down to make it on the roadmap in a 6 month chunk. It's also not too small, that some larger chunks of work can be planned. Having a second roadmap each year, also allows us to course correct mid-year. + +### Ownership + +In order to ensure things make progress, every item in the roadmap will have an owner from the project leadership team of TOC members or team maintainers. If someone really wants something on there, they will need to volunteer themselves to help keep it on track if no one else will. This will help keep the number of items on the roadmap from ballooning. The owner will also be responsible for keeping everyone up to date. All roadmap items should link to something to GitHub where things can be tracked. + +### KubeCon Alignment + +As stated above, we'll be making roadmaps with work finishing by a KubeCon event. They're coveniently around 6 months apart give or take a few weeks. As an incubation project, we're able to share project announcements at these events. This will keep the key items we're working on top of mind, and what gets finished can easily be shared. Roadmap items can also make good talk material for the maintainer track. + +With how strict travel budgets are in the current economic climate, these events are one of the few times some of us can get together. Nothing can really replace in person conversations and brainstorming. These discussions can feed into the natural recap/review of the conference and as a way to kickstart roadmap review and planning. + +## 2023H1 Roadmap + +### Release Base Image Extension +* Owner: @natalieparellano +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0105-dockerfiles.md) + +This started out as [Stack Buildpacks](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md) and now Dockerfile Extensions. Significant work has already been done on this feature over the last year. This roadmap item is about seeing this work through with releasing phase 3 in both `lifecycle` and `pack`. + +### Remove Stacks & Mixins +* Owner: @jkutner +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) + +This RFC was merged in 2021 and is a dependency on Base Image Extensions. In order to get us to 1.0, we'll need to take on some of these painful backwards breaking changes in the best way possible. This work will include the Buildpack & Platform spec changes with support in `lifecycle` and `pack`. + +### Execution Environments RFC +* Owner: @hone +* Links: Coming Soon + +There has long been a desire for a "Test" support, but it's never been prioritized even though it's made the roadmap before. Not to be over ambitious, the first step is to get a RFC written and accepted. + +### Project Health +* Owner: @samj1912 +* Links: [kpack RFC](https://github.com/buildpacks/rfcs/pull/235) + +Like other [CNCF projects](https://github.com/cncf/toc/issues?q=is%3Aissue+sort%3Aupdated-desc+%22health+of%22+-label%3A%22project+onboarding%22+-label%3A%22sandbox%22+), the project has been impacted by the VMware + Broadcom acquisition. The goal of this item is to improve the general health of the project and grow contributors back to our 2020 numbers. This inludes every team having a set of active set of maintainers and contributors, thus removing the TOC needing to step in for platform. + +As for concrete items to be accomplished: + +* Establish a buildpacks-community to be used as a labs/staging area to help hype up experiments that we would be wary of investing in otherwise and if they succeed provide them authenticity in the main buildpacks org. +* Participate in mentorship programs to grow contributors like [GSoC](https://summerofcode.withgoogle.com/) and [LFX Mentorship](https://lfx.linuxfoundation.org/tools/mentorship/). + +### Pack Test Cleaning/Optimizations +* Owner: @dfreilich +* Links: [Pack Pull Request](https://github.com/buildpacks/pack/pull/1498) + +Currently, the pack acceptance tests are very complex for newcomers. In order to help with contributions, we can relax some of these tests. + +# How it Works +[how-it-works]: #how-it-works + +See [What it is](#what-it-is) for the bulk of the details. For implementing this plan: + +* Open a PR against the [community repo](https://github.com/buildpacks/community) replacing the `ROADMAP.md`. +* As part of the regular leadership meetings we will hold check ins. +* After each KubeCon there will be a recap session and kicking off the next roadmap planning. +* In 2024, we will review how this compares to the normal annual roadmap we've traditionally done. + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + +- This will be more work with twice the number of roadmap plannings. +- There will be more overhead to ensure accountability. + +# Alternatives +[alternatives]: #alternatives + +## Do Nothing + +We can continue to do the annual roadmap. This hasn't proved very successful, but we can still take the lessons learned and adjust the current process. + +# Prior Art +[prior-art]: #prior-art + +- [Buildpacks 2021 Roadmap](https://github.com/buildpacks/community/pull/72) +- [Rust 2021 Roadmap](https://blog.rust-lang.org/2020/09/03/Planning-2021-Roadmap.html) +- [TypeScript Roadmap](https://github.com/microsoft/TypeScript/wiki/Roadmap) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +N/A + +# History +[history]: #history + + From 78fecd8af0a34c5ab4024e9fccb3b5a22868edbe Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 25 Jan 2023 18:54:14 -0600 Subject: [PATCH 028/222] update scoping on roadmap RFC Signed-off-by: Terence Lee --- text/0000-2023H1-roadmap.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/text/0000-2023H1-roadmap.md b/text/0000-2023H1-roadmap.md index 0e0684e4f..1054db0f2 100644 --- a/text/0000-2023H1-roadmap.md +++ b/text/0000-2023H1-roadmap.md @@ -48,6 +48,10 @@ For 2023, I want to propose trying something different this time around. The pro Going forward, the roadmap will account for 6 months (in this case it's just under 4 months) worth of work. This forces items on the roadmap to be smaller and concrete since they have to be something that can be accomplished in that time frame. While it was nice having the large items, they were hard to execute and it wasn't clear what the finished state looked like. Any larger piece work will need to broken down to make it on the roadmap in a 6 month chunk. It's also not too small, that some larger chunks of work can be planned. Having a second roadmap each year, also allows us to course correct mid-year. +Not only are individual items smaller, but we as a project should commit to less so they can be accomplished aren't just a bunch of empty promises. This means hard decisions will need to be made to cut highly requested features from the roadmap. + +Since this is an OSS project, things that don't make the roadmap can still be pursued by others and welcome! With finite time of maintainers, there still may be limited support depending on the maintainer. Also, items can also bubble up during the next roadmap cycle and people should advocate for them in the GitHub discussion or slack. + ### Ownership In order to ensure things make progress, every item in the roadmap will have an owner from the project leadership team of TOC members or team maintainers. If someone really wants something on there, they will need to volunteer themselves to help keep it on track if no one else will. This will help keep the number of items on the roadmap from ballooning. The owner will also be responsible for keeping everyone up to date. All roadmap items should link to something to GitHub where things can be tracked. @@ -133,7 +137,7 @@ We can continue to do the annual roadmap. This hasn't proved very successful, bu # Unresolved Questions [unresolved-questions]: #unresolved-questions -N/A +- How many items should any one person be able to own? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 49de41c21b897b07157fbbb9a5a9a4ade5a5df08 Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Thu, 26 Jan 2023 18:48:04 +0000 Subject: [PATCH 029/222] Add Buildpacks community organization RFC Signed-off-by: Sambhav Kothari --- text/0000-buildpacks-community.md | 156 ++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 text/0000-buildpacks-community.md diff --git a/text/0000-buildpacks-community.md b/text/0000-buildpacks-community.md new file mode 100644 index 000000000..8d159b674 --- /dev/null +++ b/text/0000-buildpacks-community.md @@ -0,0 +1,156 @@ +# Meta +[meta]: #meta +- Name: Buildpacks Community +- Start Date: 2023-01-26 +- Author(s): [@samj1912](https://github.com/samj1912) +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +The Buildpacks Community is a vendor-neutral Github organization where trusted community provided Cloud Native Buildpacks tooling, platforms and integrations can live. This would provide users a trusted place to search for Buildpack integrations maintained by the community. + + +# Definitions +[definitions]: #definitions + +- **Buildpacks Community** - The Buildpacks Community is a vendor-neutral Github organization where trusted community provided Cloud Native Buildpacks tooling, platforms and integrations can live. +- **Buildpacks Leadership** - The Buildpacks Leadership is a group of trusted individuals who are responsible for the Buildpacks Community. They have the ability to create repositories in the Buildpacks Community and approve new projects to be added to the Buildpacks Community. This will consist of the TOC and Team-leads of the Buildpacks project. +- **Buildpacks TOC** - The Buildpacks TOC is the technical oversight committee for the Buildpacks project. The TOC is responsible for the technical direction of the Buildpacks project. + + +# Motivation +[motivation]: #motivation + +There are two reasons why this community should exist. + +- The Buildpacks Community will allow for the testing of new technologies or the development of integrations in an environment that is more flexible than that of the core Buildpacks organization. This will provide a staging area for integrations that the Buildpacks team deems important but we are not yet ready to commit to long term maintanance. + +- A trusted repository of community integrations will also allow for a trusted source of integrations that solve common yet still relatively niche problems that are not suitable to be added to core Buildpacks organization. This will highlight integrations of high-quality and provide a vendor-neutral umbrella for them to live. They will also benefit from improved CI/CD resources and a common governance model. + +# What it is +[what-it-is]: #what-it-is + + + +For a project to be admitted to the Buildpacks community organization, it must meet the following criteria: + +- The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. +- The project must be open source and licensed under Apache 2.0. +- It must follow the Cloud Native Computing Foundation Code of Conduct. +- The project must enable DCO signoff for all commits. +- The project must be open to contributions and have a public issue tracker. +- The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. +- The list of project maintainers must be publicly available and controlled through a Github team. +- The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. +- All project contributors must be members of the Buildpacks community organization. +- The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). +- There should have visible automated testing for all repositories that are part of the project. +- The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. +- The project should strive have the following community health files: + - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. + - DEVELOPMENT.md: A guide to how contributors should develop the project. + - ADOPTERS.md: A list of adopters of the project. + - VERSIONING.md: A guide to how versioning is done for the project. + - RELEASE.md: A guide to how releases are done for the project. + - SECURITY.md: A guide to how security vulnerabilities should be reported. + +This criteria is meant to alleviate the following problems: + +- All projects must meet some testing standard to be trusted in order to ensure that the projects support the latest Buildpacks APIs and are actively maintained. +- All projects must have a clearly defined governance model to ensure that the project maintainers are elected from the community and that the project is open to contributions. +- There must be a defined system in place to reap abandonware. +- If a project maintainers are not making a best effort of patching out or updating vulnerable software then the project as a whole is untrustworthy. + + +# How it Works +[how-it-works]: #how-it-works + +## Project Admission + +A project can be admitted to the Buildpacks community organization by creating a Github issue in the Buildpacks community repository. The issue should contain the following information: + +- Name of the project +- Evidence for the above criteria +- A list of maintainers for the project + +The above information will be structured into an appropriate issue template. The Buildpacks Leadership will review the issue and if the project meets the above criteria, the project will be added to the Buildpacks community organization. The Buildpacks Leadership will assign a steward team to the project and the team lead of the steward team will be responsible for ensuring that the project meets the above criteria. + +Once admitted, the team lead of the steward team will create a Github team for the project and add the project maintainers to the team and mark them as the team maintainers allowing them to add other maintainers. The existing team maintainers of the steward team will be added as maintainers to the project team. + +The team lead will also create a CODEOWNERS file for the project and add the project maintainers as the code owners. + +The project maintainers will be responsible for maintaining the list of project maintainers and ensuring that all project contributors are members of the Buildpacks community organization. + +They will be able to add new Github members to the organization by creating a Github issue in the Buildpacks community [invites repository](https://github.com/buildpacks-community/invites). + +## Project Removal + +In case the project fails to meet the above criteria, the Buildpacks Leadership will work with the project maintainers to address the issues and if the project is still not ready, the project will be archived or removed from the Buildpacks community organization. + +## Project Graduation + +In case a project is deemed to be mature enough to be part of the core Buildpacks organization, the project maintainers can request for the project to be graduated to the core Buildpacks organization via the [Component Contribution RFC](https://github.com/buildpacks/rfcs/blob/main/text/0108-governance-component-maintainer-role.md). The Buildpacks TOC will review the request and if the project meets the criteria for graduation, the project will be moved to the core Buildpacks organization. + + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + + +N/A + +# Alternatives +[alternatives]: #alternatives + +N/A + +# Prior Art +[prior-art]: #prior-art + +- [CNCF Sandbox](https://www.cncf.io/sandbox-projects/) +- [Paketo Community](https://github.com/paketo-buildpacks/rfcs/blob/main/text/0008-paketo-community.md) +- [Argoproj Labs](https://github.com/argoproj-labs) +- [Crossplane Contrib](https://github.com/crossplane-contrib) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- The exact project onboarding issue template. + +# History +[history]: #history + + + From 1aa20776e8e4eb9939449d8806f309a2038bb6fe Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Fri, 27 Jan 2023 23:39:27 +0000 Subject: [PATCH 030/222] Update text/0000-buildpacks-community.md Co-authored-by: Natalie Arellano Signed-off-by: Sambhav Kothari --- text/0000-buildpacks-community.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-buildpacks-community.md b/text/0000-buildpacks-community.md index 8d159b674..44d1039cb 100644 --- a/text/0000-buildpacks-community.md +++ b/text/0000-buildpacks-community.md @@ -85,7 +85,7 @@ A project can be admitted to the Buildpacks community organization by creating a - Evidence for the above criteria - A list of maintainers for the project -The above information will be structured into an appropriate issue template. The Buildpacks Leadership will review the issue and if the project meets the above criteria, the project will be added to the Buildpacks community organization. The Buildpacks Leadership will assign a steward team to the project and the team lead of the steward team will be responsible for ensuring that the project meets the above criteria. +The above information will be structured into an appropriate issue template. The Buildpacks Leadership will review the issue and if the project meets the above criteria, the project will be added to the Buildpacks community organization. The Buildpacks Leadership will assign a team to the project and the team lead of the team will steward the project - i.e., will be responsible for ensuring that the project meets the above criteria. Once admitted, the team lead of the steward team will create a Github team for the project and add the project maintainers to the team and mark them as the team maintainers allowing them to add other maintainers. The existing team maintainers of the steward team will be added as maintainers to the project team. From c6755535c9ffe99698b4b150efed3cc3c78f235b Mon Sep 17 00:00:00 2001 From: Aidan Delaney Date: Tue, 31 Jan 2023 06:01:57 +0000 Subject: [PATCH 031/222] Update text/0000-2023H1-roadmap.md Co-authored-by: Natalie Arellano Signed-off-by: Aidan Delaney --- text/0000-2023H1-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-2023H1-roadmap.md b/text/0000-2023H1-roadmap.md index 1054db0f2..d78d147f9 100644 --- a/text/0000-2023H1-roadmap.md +++ b/text/0000-2023H1-roadmap.md @@ -38,7 +38,7 @@ As an incubation level project, there is an opportunity to broadcast announcemen ## Roadmap Changes -For 2023, I want to propose trying something different this time around. The project will publish two roadmaps one for each half of the year aligned with work to be completed for KubeCon EU (2023-05-16) and KubeCon NA (2023-10-24). The goals I'm hoping to see: +For 2023, I want to propose trying something different this time around. The project will publish two roadmaps one for each half of the year aligned with work to be completed for KubeCon EU (2023-04-18) and KubeCon NA (2023-11-6). The goals I'm hoping to see: - Focus - Decrease the amount we're doing as a project, so we're able to deliver on the things we commit to. - Accountability - With that focus, we also need to be accountable for what we're putting on this list. From 36f5d9d25fdd8255fffefcca6e7c666818e04a99 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Tue, 31 Jan 2023 18:02:07 -0600 Subject: [PATCH 032/222] add Buildpacks Community Organization RFC Signed-off-by: Terence Lee --- text/0000-2023H1-roadmap.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-2023H1-roadmap.md b/text/0000-2023H1-roadmap.md index d78d147f9..e82d4644a 100644 --- a/text/0000-2023H1-roadmap.md +++ b/text/0000-2023H1-roadmap.md @@ -3,7 +3,7 @@ - Name: 2023H1 Roadmap - Start Date: 2023-01-24 - Author(s): hone -- Status: Draft Draft +- Status: Draft - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) @@ -84,7 +84,7 @@ There has long been a desire for a "Test" support, but it's never been prioritiz ### Project Health * Owner: @samj1912 -* Links: [kpack RFC](https://github.com/buildpacks/rfcs/pull/235) +* Links: [Buildpacks Community Organization RFC](https://github.com/buildpacks/rfcs/pull/273) Like other [CNCF projects](https://github.com/cncf/toc/issues?q=is%3Aissue+sort%3Aupdated-desc+%22health+of%22+-label%3A%22project+onboarding%22+-label%3A%22sandbox%22+), the project has been impacted by the VMware + Broadcom acquisition. The goal of this item is to improve the general health of the project and grow contributors back to our 2020 numbers. This inludes every team having a set of active set of maintainers and contributors, thus removing the TOC needing to step in for platform. From 52d7577505f71dd2f465b32068185f13fb42dea9 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 1 Feb 2023 02:17:25 -0600 Subject: [PATCH 033/222] add execution environment RFC Signed-off-by: Terence Lee --- text/0000-2023H1-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-2023H1-roadmap.md b/text/0000-2023H1-roadmap.md index e82d4644a..317a2dd94 100644 --- a/text/0000-2023H1-roadmap.md +++ b/text/0000-2023H1-roadmap.md @@ -78,7 +78,7 @@ This RFC was merged in 2021 and is a dependency on Base Image Extensions. In ord ### Execution Environments RFC * Owner: @hone -* Links: Coming Soon +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/274) There has long been a desire for a "Test" support, but it's never been prioritized even though it's made the roadmap before. Not to be over ambitious, the first step is to get a RFC written and accepted. From 51edf8dc2bafb74715faf66c32ff9255fd6b97f5 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 1 Feb 2023 07:17:28 -0800 Subject: [PATCH 034/222] Update text/0000-2023H1-roadmap.md Co-authored-by: Aidan Delaney Signed-off-by: Terence Lee --- text/0000-2023H1-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-2023H1-roadmap.md b/text/0000-2023H1-roadmap.md index 317a2dd94..2c5538926 100644 --- a/text/0000-2023H1-roadmap.md +++ b/text/0000-2023H1-roadmap.md @@ -90,7 +90,7 @@ Like other [CNCF projects](https://github.com/cncf/toc/issues?q=is%3Aissue+sort% As for concrete items to be accomplished: -* Establish a buildpacks-community to be used as a labs/staging area to help hype up experiments that we would be wary of investing in otherwise and if they succeed provide them authenticity in the main buildpacks org. +* Establish a buildpacks-community to be used as a labs/staging area to help hype up experiments that we would be otherwise wary of investing in and, if they succeed, adopt them in the main buildpacks org. * Participate in mentorship programs to grow contributors like [GSoC](https://summerofcode.withgoogle.com/) and [LFX Mentorship](https://lfx.linuxfoundation.org/tools/mentorship/). ### Pack Test Cleaning/Optimizations From a17c023c46f24fce1686cf51244123ec3adeec41 Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Wed, 1 Feb 2023 23:14:38 +0000 Subject: [PATCH 035/222] RFC 0117 [#273] Signed-off-by: Sambhav Kothari --- ...buildpacks-community.md => 0117-buildpacks-community.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-buildpacks-community.md => 0117-buildpacks-community.md} (98%) diff --git a/text/0000-buildpacks-community.md b/text/0117-buildpacks-community.md similarity index 98% rename from text/0000-buildpacks-community.md rename to text/0117-buildpacks-community.md index 44d1039cb..d92c089ab 100644 --- a/text/0000-buildpacks-community.md +++ b/text/0117-buildpacks-community.md @@ -3,10 +3,10 @@ - Name: Buildpacks Community - Start Date: 2023-01-26 - Author(s): [@samj1912](https://github.com/samj1912) -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#273](https://github.com/buildpacks/rfcs/pull/273) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: N/A - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Summary From 733281523bc822fee8d96946fcadb2b66ae66ed3 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Fri, 3 Feb 2023 10:32:18 -0500 Subject: [PATCH 036/222] Update 0078-group-additions.md Not actually implemented as per https://cloud-native.slack.com/archives/C0331B61A1Y/p1675364009281399 Signed-off-by: Natalie Arellano --- text/0078-group-additions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0078-group-additions.md b/text/0078-group-additions.md index df01c6354..9eed25872 100644 --- a/text/0078-group-additions.md +++ b/text/0078-group-additions.md @@ -3,7 +3,7 @@ - Name: Group additions to Builder order - Start Date: 2020-12-23 - Author(s): [jkutner](@jkutner) -- Status: Implemented +- Status: Approved - RFC Pull Request: [rfcs#129](https://github.com/buildpacks/rfcs/pull/129) - CNB Pull Request: (leave blank) - CNB Issue: [buildpacks/docs#319](https://github.com/buildpacks/docs/issues/319), [buildpacks/pack#1099](https://github.com/buildpacks/pack/issues/1099), [buildpacks/pack#1100](https://github.com/buildpacks/pack/issues/1100), [buildpacks/spec#195](https://github.com/buildpacks/spec/issues/195) From b95dc2ca6818bdb7ef1f363e8db131ac3fcb8078 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 23 Feb 2022 10:22:47 -0500 Subject: [PATCH 037/222] WIP - Draft version of the RFC Signed-off-by: Juan Bustamante --- text/0000-publish-operation | 121 ++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 text/0000-publish-operation diff --git a/text/0000-publish-operation b/text/0000-publish-operation new file mode 100644 index 000000000..78f14571a --- /dev/null +++ b/text/0000-publish-operation @@ -0,0 +1,121 @@ +# Meta +[meta]: #meta +- Name: Publish Operation +- Start Date: 2022-02-22 +- Author(s): Juan Bustamante (@jbustamante) +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +Split the process of generating and saving it into the final destination in two different operations. +- Export: will create the image and saving it to disk in OCI layout format. When the `daemon` flag is enabled it will save all the metadata that can't be saved into the daemon in a separate report +- Publish: will take the image generated by Export and save it into the Daemon or in the Registry using the metadata report when it's necessary. + +# Definitions +[definitions]: #definitions + + +# Motivation +[motivation]: #motivation + + + +Because the [Image.Digest](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1#Image) method from the GCR library would compute the value before pushing it to a Registry, we can be sure to create the Image and add all the metadata require (annotations, cosign, etc) save it to disk and be sure that when the image is push to the registry the Image Digest will not change. (See the [thread](https://cloud-native.slack.com/archives/C033DV9EBDF/p1644523524402149) on Slack) + +Splitting the creation of the image and it's saving operation in two different process allows us the following: +- Implement an export phase with an standard output format (OCI Layout) that can be consumed by `Platforms` if they want to +- Keep consistency with the push operation behavior because the digest and the metadata can be saved linked to the Manifest +- Solve unsupported metadata cases by the Daemon, because we can save this information in a custom format together with the Image exported in OCI Layout + + +We can support the use cases: + +- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) +- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) + + + +# What it is +[what-it-is]: #what-it-is + +The idea is to modify the existing *Export* phase to save the image on disk in a path defined by **Platform**, in case of **Pack** the image +would be saved in a volumen shared by other phases, OCI Layout forma will be used to saved the image and a new operation called *Publish* +will be created. This operation will be invoked when the a new flag `--publish` is sent to Lifecycle and it will actually load +the image from disk generated by the export phase and then push to Daemon or Registry. + +Check the following image for a visual representation of the idea. + +```mermaid +graph LR + A[ANALIZE] --> B[RESTORE] + B --> C[BUILD] + C --> D[EXPORT] + D --> |when --publish| E[PUBLISH] + +``` + + + + + +# How it Works +[how-it-works]: #how-it-works + + + +# Migration +[migration]: #migration + + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? +- Why is this proposal the best? +- What is the impact of not doing this? + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? +- What parts of the design do you expect to be resolved through implementation of the feature? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes +Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. +Examples of a spec. change might be new lifecycle flags, new `buildpack.toml` fields, new fields in the buildpackage label, etc. +This section is not intended to be binding, but as discussion of an RFC unfolds, if spec changes are necessary, they should be documented here. From ff921a6033eafe02c8f54b8eb3b9b983f1620363 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 23 Feb 2022 10:29:19 -0500 Subject: [PATCH 038/222] Fixing extension of the file Signed-off-by: Juan Bustamante --- text/{0000-publish-operation => 0000-publish-operation.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-publish-operation => 0000-publish-operation.md} (100%) diff --git a/text/0000-publish-operation b/text/0000-publish-operation.md similarity index 100% rename from text/0000-publish-operation rename to text/0000-publish-operation.md From 430ad6d13495cfcd0905cfe26f4e7f77be2db478 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 24 Feb 2022 09:30:16 -0500 Subject: [PATCH 039/222] more work Signed-off-by: Juan Bustamante --- text/0000-publish-operation.md | 65 +++++++++++++++++++++++----------- 1 file changed, 44 insertions(+), 21 deletions(-) diff --git a/text/0000-publish-operation.md b/text/0000-publish-operation.md index 78f14571a..9220b08d3 100644 --- a/text/0000-publish-operation.md +++ b/text/0000-publish-operation.md @@ -12,43 +12,58 @@ # Summary [summary]: #summary -Split the process of generating and saving it into the final destination in two different operations. -- Export: will create the image and saving it to disk in OCI layout format. When the `daemon` flag is enabled it will save all the metadata that can't be saved into the daemon in a separate report -- Publish: will take the image generated by Export and save it into the Daemon or in the Registry using the metadata report when it's necessary. +Split the process of creating and saving the Image in two different operations. +- Export: will create the Image and saves it to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. When the `daemon` flag is enabled it could save all the metadata that can't be saved into the daemon in a separate report. +- Publish: will take the image generated by Export and push it into the Daemon or in the Registry. # Definitions [definitions]: #definitions - + +- A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. +- A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. +- A Daemon is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. +- A Registry is a long-running service used for storing and retrieving container images. +- An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and location-addressable references. +- An [image index](https://github.com/opencontainers/image-spec/blob/main/image-index.md) is a higher-level manifest which points to a list of manifests and descriptors. +- An [Image manifest](https://github.com/opencontainers/image-spec/blob/main/manifest.md) provides a configuration and set of layers for a single container image for a specific architecture and operating system. +- A [config](https://github.com/opencontainers/image-spec/blob/main/descriptor.md) is a property references a configuration object for a container, by digest. It must support the following media type `application/vnd.oci.image.config.v1+json` # Motivation [motivation]: #motivation +Because the [Image.Digest](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1#Image) method from the GCR library would compute the Digest value before pushing it to a Registry, it allows us to create the image and add metadata require like annotations or even sign it and save it to disk and be sure that when the Image is push to the registry the Digest will not change. (See the [thread](https://cloud-native.slack.com/archives/C033DV9EBDF/p1644523524402149) on Slack). + + + +We should do this to unblock uses cases like + +- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) +- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) + + + + +We expect the `Export` phase to save the Image in disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and the `Publish` operation to push it to Daemon or Registry. + + -We can support the use cases: - -- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) -- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) +--> - # What it is [what-it-is]: #what-it-is -The idea is to modify the existing *Export* phase to save the image on disk in a path defined by **Platform**, in case of **Pack** the image -would be saved in a volumen shared by other phases, OCI Layout forma will be used to saved the image and a new operation called *Publish* -will be created. This operation will be invoked when the a new flag `--publish` is sent to Lifecycle and it will actually load -the image from disk generated by the export phase and then push to Daemon or Registry. +An new operation called *Publish* will be added in the Lifecycle. `/cnb/lifecycle/publisher` is responsibly for pushing the Image into the Daemon or into an OCI Registry. + + +The *Export* phase will be modify to save the image on disk in a path defined by **Platform**, in case of **Pack**, for example, the image would be saved in a shared volume in the same way it's done right now, [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be used to save the Image on disk. Check the following image for a visual representation of the idea. @@ -57,12 +72,10 @@ graph LR A[ANALIZE] --> B[RESTORE] B --> C[BUILD] C --> D[EXPORT] - D --> |when --publish| E[PUBLISH] + D --> |when --push| E[PUBLISH] ``` - - # How it Works From 945b9869ef01f1f42f0f6e29a03a68dbb17da08e Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 24 Feb 2022 13:46:07 -0500 Subject: [PATCH 040/222] WIP - adding more diagrams Signed-off-by: Juan Bustamante --- text/0000-publish-operation.md | 65 +++++++++++++++++++++++++++------- 1 file changed, 53 insertions(+), 12 deletions(-) diff --git a/text/0000-publish-operation.md b/text/0000-publish-operation.md index 9220b08d3..c98ee64a1 100644 --- a/text/0000-publish-operation.md +++ b/text/0000-publish-operation.md @@ -12,9 +12,9 @@ # Summary [summary]: #summary -Split the process of creating and saving the Image in two different operations. -- Export: will create the Image and saves it to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. When the `daemon` flag is enabled it could save all the metadata that can't be saved into the daemon in a separate report. -- Publish: will take the image generated by Export and push it into the Daemon or in the Registry. +Split the process of creating and writing the Image in a final destination in two different operations. +- Export: will create the Image and saves it to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +- Publish: will take the Image generated by Export and write it into the Daemon or in the Registry. # Definitions [definitions]: #definitions @@ -60,19 +60,57 @@ We expect the `Export` phase to save the Image in disk in [OCI Image Layout](htt # What it is [what-it-is]: #what-it-is -An new operation called *Publish* will be added in the Lifecycle. `/cnb/lifecycle/publisher` is responsibly for pushing the Image into the Daemon or into an OCI Registry. +Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to change that behavior to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. Then a new operation called *Publish* will be added in the Lifecycle. `/cnb/lifecycle/publisher` is responsibly for writing the Image either in a OCI image registry or a docker daemon. - -The *Export* phase will be modify to save the image on disk in a path defined by **Platform**, in case of **Pack**, for example, the image would be saved in a shared volume in the same way it's done right now, [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be used to save the Image on disk. - -Check the following image for a visual representation of the idea. +The new Lifecycle orchestration process will change to something like this: ```mermaid graph LR - A[ANALIZE] --> B[RESTORE] - B --> C[BUILD] - C --> D[EXPORT] - D --> |when --push| E[PUBLISH] + A[DETECT] --> B[ANALYZE] + B --> C[RESTORE] + C --> D[BUILD] + D --> E[EXPORT] + E --> |when --push| F[PUBLISH] + +``` + +This features affects the Platform implementor because they will have to include this new operation as part of the workflow to build the application image. + +Let's see some examples: + +## Examples + +### Using the Creator + +#### Pushing an image to daemon + +```=shell + /cnb/lifecycle/creator -daemon -push + +``` + +#### Pushing an image to an OCI registry + +```=shell + /cnb/lifecycle/creator -push + +``` + +### Not using the Creator + +#### Pushing an image to daemon + +```=shell + /cnb/lifecycle/exporter -output /path/to/save/the/image + /cnb/lifecycle/publisher -daemon -input /path/to/the/image [...] + +``` + +#### Pushing an image to an OCI registry + +```=shell + /cnb/lifecycle/exporter -output /path/to/save/the/image + /cnb/lifecycle/publisher -input /path/to/the/image [...] ``` @@ -100,6 +138,9 @@ When it's invoke from the Creator the current workflow is not affected because a # How it Works [how-it-works]: #how-it-works + +![](https://i.imgur.com/DLMMpGf.png) + +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +When the `Exporter` phase is invoked passing the `-daemon` flag besides writing into the Daemon also save the image to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format updating the [report.toml](https://github.com/buildpacks/spec/blob/main/platform.md#reporttoml-toml) file with all the metadata require to validate the consistency of the image when it is published to a Registry. + +# Definitions +[definitions]: #definitions + +- A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. +- A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. +- A Daemon is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. +- A Registry is a long-running service used for storing and retrieving container images. +- A digest reference refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form /@ which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A Image Manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. +- An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and location-addressable references. + +# Motivation +[motivation]: #motivation + +Implementing this new feature will help us to solve the problem of loosing information when the image is saved into the Daemon keeping the image on disk along with the metadata it can be used as input for other tools to offer more capabilities to the end users. + +This feature will help to unblock uses cases like +- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) +- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) + +# What it is +[what-it-is]: #what-it-is + +Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to add the capability to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format only when the `-daemon` flag is used as argument AND the feature is enable using a new flag `-layout` or the default environment variable `CNB_LAYOUT_DIR`. + +## Examples + +### Exporting using the environment variable + +```=shell +> export CNB_LAYOUT_DIR=oci +> /cnb/lifecycle/exporter -daemon my-app-image +> tree /oci +. +└── oci/ + └── my-app-image/ + ├── blobs/ + │ └── sha256/ + │ └── 01.. + ├── index.json + └── oci-layout + +``` + +### Exporting using the command line flag + +```=shell +> /cnb/lifecycle/exporter -daemon -layout oci my-app-image +> tree /oci +. +└── oci/ + └── my-app-image/ + ├── blobs/ + │ └── sha256/ + │ └── 01.. + ├── index.json + └── oci-layout + +``` +As we can see there is a new folder called `oci` and inside that folder we can find our application image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format + +Attempts to use this feature when the `-daemon` flag is not used could be ignore or show some warnings messages. + +# How it Works +[how-it-works]: #how-it-works + +The lifecycle phases affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) + +The following new input is proposed to be added to this phase + +| Input | Environment Variable | Default Value | Description +|-------------------|-----------------------|--------------------------|---------------------- +| `` | `CNB_LAYOUT_DIR` | "" | The root directory where the OCI image will be written. The presence of a none empty value for this environment variable will enable the feature. | + + +- When the exporter is executed WITH the flag `daemon` it will check for the presence of the flag `layout` or the environment variable `CNB_LAYOUT_DIR` +- IF any of the values are present THEN + - It will create a folder name `` located at the path defined `` or `CNB_LAYOUT_DIR` + - In parallel of exporting the Image to the Daemon it will save the final Image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format in the directory created before. The image will be saved using **uncompressed** layers + - It will calculate the digest of the manifest of the compressed layers and write that value into the report.toml file + - It will update the report.toml file with all the tags and require information to verify the image once it is pushed into a registry +- OTHERWISE it will behave as it is right now + +# Migration +[migration]: #migration + + +# Drawbacks +[drawbacks]: #drawbacks + +- We could increase the disk space if we not managed the duplication of saving the layers on disk. Currently the Cache implementation (used when daemon is ON) saved the layers tarballs on disk, because the current proposal is exporting the whole image on disk it will also require more space to save the layers for the OCI format in the `blobs` folder. + +# Alternatives +[alternatives]: #alternatives + +## Redesign the current Cache + +Another potential solution could be to export the OCI image along with the current Cache implementation. The current implementation when the Daemon is enable can be describe with the following class diagram + +```mermaid +classDiagram + Cache <|-- VolumeCache + Image <|-- LocalImage + LocalImage <|-- CachingImage + CachingImage .. VolumeCache +class Cache { + <> +} + +class Image { + <> +} +``` + +When the Daemon is enabled, a `CachingImage` is created, this image implementation saves the layers tarballs in a `VolumeCache` to reuse them and increase the speed of the process. The idea could be to redesign this `Cache` implementation (VolumeCache) with a new one, maybe a `OCICache` which handles the details to avoid saving the layers tarballs duplicated. + +```mermaid +classDiagram + Cache <|-- OCICache + Image <|-- LocalImage + LocalImage <|-- CachingImage + CachingImage .. OCICache +class Cache { + <> + +} + +class Image { + <> + +} +``` +### Drawbacks + +- We will have to expose Cache implementation details to the outside world, probably spec those details, for other tools to interact with this exported data + + + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + + diff --git a/text/0000-publish-operation.md b/text/0000-publish-operation.md deleted file mode 100644 index c98ee64a1..000000000 --- a/text/0000-publish-operation.md +++ /dev/null @@ -1,185 +0,0 @@ -# Meta -[meta]: #meta -- Name: Publish Operation -- Start Date: 2022-02-22 -- Author(s): Juan Bustamante (@jbustamante) -- Status: Draft -- RFC Pull Request: (leave blank) -- CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) -- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) - -# Summary -[summary]: #summary - -Split the process of creating and writing the Image in a final destination in two different operations. -- Export: will create the Image and saves it to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -- Publish: will take the Image generated by Export and write it into the Daemon or in the Registry. - -# Definitions -[definitions]: #definitions - -- A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. -- A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. -- A Daemon is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. -- A Registry is a long-running service used for storing and retrieving container images. -- An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and location-addressable references. -- An [image index](https://github.com/opencontainers/image-spec/blob/main/image-index.md) is a higher-level manifest which points to a list of manifests and descriptors. -- An [Image manifest](https://github.com/opencontainers/image-spec/blob/main/manifest.md) provides a configuration and set of layers for a single container image for a specific architecture and operating system. -- A [config](https://github.com/opencontainers/image-spec/blob/main/descriptor.md) is a property references a configuration object for a container, by digest. It must support the following media type `application/vnd.oci.image.config.v1+json` - -# Motivation -[motivation]: #motivation - -Because the [Image.Digest](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1#Image) method from the GCR library would compute the Digest value before pushing it to a Registry, it allows us to create the image and add metadata require like annotations or even sign it and save it to disk and be sure that when the Image is push to the registry the Digest will not change. (See the [thread](https://cloud-native.slack.com/archives/C033DV9EBDF/p1644523524402149) on Slack). - - - - -We should do this to unblock uses cases like - -- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) -- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) - - - - -We expect the `Export` phase to save the Image in disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and the `Publish` operation to push it to Daemon or Registry. - - - - -# What it is -[what-it-is]: #what-it-is - -Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to change that behavior to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. Then a new operation called *Publish* will be added in the Lifecycle. `/cnb/lifecycle/publisher` is responsibly for writing the Image either in a OCI image registry or a docker daemon. - -The new Lifecycle orchestration process will change to something like this: - -```mermaid -graph LR - A[DETECT] --> B[ANALYZE] - B --> C[RESTORE] - C --> D[BUILD] - D --> E[EXPORT] - E --> |when --push| F[PUBLISH] - -``` - -This features affects the Platform implementor because they will have to include this new operation as part of the workflow to build the application image. - -Let's see some examples: - -## Examples - -### Using the Creator - -#### Pushing an image to daemon - -```=shell - /cnb/lifecycle/creator -daemon -push - -``` - -#### Pushing an image to an OCI registry - -```=shell - /cnb/lifecycle/creator -push - -``` - -### Not using the Creator - -#### Pushing an image to daemon - -```=shell - /cnb/lifecycle/exporter -output /path/to/save/the/image - /cnb/lifecycle/publisher -daemon -input /path/to/the/image [...] - -``` - -#### Pushing an image to an OCI registry - -```=shell - /cnb/lifecycle/exporter -output /path/to/save/the/image - /cnb/lifecycle/publisher -input /path/to/the/image [...] - -``` - - - -# How it Works -[how-it-works]: #how-it-works - - -![](https://i.imgur.com/DLMMpGf.png) - - - -# Migration -[migration]: #migration - - -# Drawbacks -[drawbacks]: #drawbacks - -Why should we *not* do this? - -# Alternatives -[alternatives]: #alternatives - -- What other designs have been considered? -- Why is this proposal the best? -- What is the impact of not doing this? - -# Prior Art -[prior-art]: #prior-art - -Discuss prior art, both the good and bad. - -# Unresolved Questions -[unresolved-questions]: #unresolved-questions - -- What parts of the design do you expect to be resolved before this gets merged? -- What parts of the design do you expect to be resolved through implementation of the feature? -- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? - -# Spec. Changes (OPTIONAL) -[spec-changes]: #spec-changes -Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. -Examples of a spec. change might be new lifecycle flags, new `buildpack.toml` fields, new fields in the buildpackage label, etc. -This section is not intended to be binding, but as discussion of an RFC unfolds, if spec changes are necessary, they should be documented here. From 7150fa1cd475fab3673d3308433b1aa82cda1791 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 2 Mar 2022 11:04:14 -0500 Subject: [PATCH 042/222] WIP - Adding some C4 models diagrams to complement the idea Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 8d30d551d..37095b2db 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -82,6 +82,12 @@ Attempts to use this feature when the `-daemon` flag is not used could be ignore The lifecycle phases affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) +At high level view of the propose solution can be summarize with the following container diagram from the C4 model + +![](https://i.imgur.com/XhtvT9j.png) + +Notice that we are relying on the OCI format Specification to expose the data for `Platforms` + The following new input is proposed to be added to this phase | Input | Environment Variable | Default Value | Description @@ -97,6 +103,8 @@ The following new input is proposed to be added to this phase - It will update the report.toml file with all the tags and require information to verify the image once it is pushed into a registry - OTHERWISE it will behave as it is right now + + # Migration [migration]: #migration @@ -113,7 +121,13 @@ This section should document breaks to public API and breaks in compatibility du ## Redesign the current Cache -Another potential solution could be to export the OCI image along with the current Cache implementation. The current implementation when the Daemon is enable can be describe with the following class diagram +Another potential solution could be to export the OCI image along with the current Cache concept and expose some contract for `Platform` to interact with this Cache and extract the final OCI image. At high level, the solution can be represented with the following container diagram from C4 model + +![](https://i.imgur.com/xl4gL1G.png) + +Notice that on this solution, because the Cache is an internal component from the Lifecycle implementation we will have to expose some kind of specification for `Platforms` to understand its format and been able to read the OCI image. + +The current implementation when the Daemon is enable can be describe with the following class diagram ```mermaid classDiagram From b56954e71887484b3c6660c1d89f68eda35d6004 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 3 Mar 2022 09:48:30 -0500 Subject: [PATCH 043/222] WIP - adding a draft idea of some changes to the report.toml file Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 37095b2db..a85869263 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -103,7 +103,28 @@ The following new input is proposed to be added to this phase - It will update the report.toml file with all the tags and require information to verify the image once it is pushed into a registry - OTHERWISE it will behave as it is right now - +#### `report.toml` (TOML) + +The new information to be added into the `report.toml` file can be summarize as follows: + +```toml +[export] +[[export.oci]] +digest = "" +manifest-size = "" +compression-algorithm = "" +library-url = "https://github.com/google/go-containerregistry/" +library-language="go" +``` +Where: +- **If** the app image was exported using the `-layer` flag, the export section will be added to the report + - `digest` MUST contain the image digest calculated based on compressed layers + - `manifest-size` MUST contain the manifest size in bytes + - `compression-algorithm` MUST contain the information of the algorithm used by GCR to compute the digest + - `library-url` MUST contain the url of the library used to export the image + - `library-language` MUST contain the client's library programming language used to export the image + +The main idea of this new section in the `report.toml` file is to provide the information of the **expected** digest of the image if it is exported to a registry using a particular implementation. This information can be used by other tools (like publish) to complete the operation and verify the consistency of the image. # Migration [migration]: #migration From 49c555cca4a8a8b4b6833a3e534c2cb5587a24f3 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 4 Mar 2022 16:29:16 -0500 Subject: [PATCH 044/222] Apply suggestions from code review Co-authored-by: Natalie Arellano Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index a85869263..83cca2bd8 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -140,13 +140,13 @@ This section should document breaks to public API and breaks in compatibility du # Alternatives [alternatives]: #alternatives -## Redesign the current Cache +## Redesign the current Launch Cache -Another potential solution could be to export the OCI image along with the current Cache concept and expose some contract for `Platform` to interact with this Cache and extract the final OCI image. At high level, the solution can be represented with the following container diagram from C4 model +Another potential solution could be to export the OCI image along with the current Launch Cache concept and expose some contract for `Platform` to interact with this Cache and extract the final OCI image. At high level, the solution can be represented with the following container diagram from C4 model ![](https://i.imgur.com/xl4gL1G.png) -Notice that on this solution, because the Cache is an internal component from the Lifecycle implementation we will have to expose some kind of specification for `Platforms` to understand its format and been able to read the OCI image. +Notice that on this solution, because the Launch Cache is an internal component from the Lifecycle implementation we will have to expose some kind of specification for `Platforms` to understand its format and been able to read the OCI image. The current implementation when the Daemon is enable can be describe with the following class diagram From 480896f6cbf09479c4b5694a70fc7905d62779ec Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 4 Mar 2022 16:55:16 -0500 Subject: [PATCH 045/222] Apply suggestions from code review Co-authored-by: Natalie Arellano Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 83cca2bd8..9c4fd1535 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -185,7 +185,7 @@ class Image { ``` ### Drawbacks -- We will have to expose Cache implementation details to the outside world, probably spec those details, for other tools to interact with this exported data +- We will have to expose Launch Cache implementation details to the outside world, probably spec those details, for other tools to interact with this exported data |Yes| B + A -->|No| END + B{IS -launch-cache defined?} -->|Yes|D + B -->|No| E + E{DOES layout-dir/image exists?} --> |Yes| L + L[...] + E --> |No| M + M[Create layout-dir/image directory] --> O[export-dir = layout-dir/image] + O --> I + D[/Warn: will export to launch cache dir/] --> F + F{DOES launch-cache/image dir exists?} -->|Yes| G + G[ ...] + F -->|No| H + H[Create launch-cache/image directory] --> N[export-dir = launch-cache/image] + N --> I[Write image to $export-dir in OCI format **] + I --> J[Calculate manifest's digest] + J --> K[/Write digest into report.toml/] + K -->END((End)) + +``` + +Notes: + - WHEN `-launch-cache` flow is executed + - The content of `blobs//` MAY contain symbolic links to content saved in the launch cache to avoid duplicating files. + - The content of `blobs//` MAY reference tar files in **uncompressed** format because that's how they are saved in the cache + - WHEN `-launch-cache` IS NOT defined + - The content of `blobs//` MAY be saved in **compressed** format + #### `report.toml` (TOML) From 6ab321df0a0ae2d64e8daf8073644ea6f7dacf6f Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 10 Mar 2022 17:43:36 -0500 Subject: [PATCH 048/222] WIP - formating issues Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index f4d92f625..6007980fc 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -166,22 +166,28 @@ The following new input is proposed to be added to this phase |-------------------|-----------------------|--------------------------|---------------------- | `` | `CNB_LAYOUT_DIR` | "" | The root directory where the OCI image will be written. The presence of a none empty value for this environment variable will enable the feature. | - -The following flowchart explains the flow proposed +Let's see the propose flow ```mermaid flowchart - A{IS -layout OR CNB_LAYOUT_DIR defined?} -->|Yes| B + A{"IS -layout OR + CNB_LAYOUT_DIR + defined?"} -->|Yes| B A -->|No| END - B{IS -launch-cache defined?} -->|Yes|D + B{"IS -launch-cache + defined?"} -->|Yes|D B -->|No| E - E{DOES layout-dir/image exists?} --> |Yes| L + E{"DOES + layout-dir/image + exist?"} --> |Yes| L L[...] E --> |No| M M[Create layout-dir/image directory] --> O[export-dir = layout-dir/image] O --> I D[/Warn: will export to launch cache dir/] --> F - F{DOES launch-cache/image dir exists?} -->|Yes| G + F{"DOES + launch-cache/image + dir exist?"} -->|Yes| G G[ ...] F -->|No| H H[Create launch-cache/image directory] --> N[export-dir = launch-cache/image] @@ -189,7 +195,6 @@ flowchart I --> J[Calculate manifest's digest] J --> K[/Write digest into report.toml/] K -->END((End)) - ``` Notes: From 710ed11b08044fe0feb74d24b3abcb34f53486e8 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 16 Mar 2022 16:46:23 -0500 Subject: [PATCH 049/222] Improvements to the flowchart - Adding feedback from review - Simplify the flowchart diagram to avoid duplicated steps - Added the pending flows in the flowchart diagram Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 52 ++++++++++++++++++++------------------ 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 6007980fc..6e0bd8d41 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -63,7 +63,7 @@ Or > /cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -layout /oci my-app-image ``` -The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/launch-cache/my-app-image/` folder. +The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/launch-cache/oci/my-app-image/` folder. ```=shell > cd /launch-cache @@ -76,14 +76,15 @@ The expected output is the `my-app-image` exported in [OCI Image Layout](https:/ │ ├── sha256:6905011516dcf4...tar │ └── sha256:83d85471d9f8a3...tar ├── staging - └── my-app-image/ - ├── blobs/ - │ └── sha256/ - │ ├── 65d9067f915e01...tar -> /launch-cache/committed/sha256:65d9067f915e01...tar - │ ├── 6905011516dcf4...tar -> /launch-cache/committed/sha256:6905011516dcf4...tar - │ └── 83d85471d9f8a3...tar -> /launch-cache/committed/sha256:83d85471d9f8a3...tar - ├── index.json - └── oci-layout + └── oci/ + └── my-app-image/ + ├── blobs/ + │ └── sha256/ + │ ├── 65d9067f915e01...tar -> /launch-cache/committed/sha256:65d9067f915e01...tar + │ ├── 6905011516dcf4...tar -> /launch-cache/committed/sha256:6905011516dcf4...tar + │ └── 83d85471d9f8a3...tar -> /launch-cache/committed/sha256:83d85471d9f8a3...tar + ├── index.json + └── oci-layout ``` @@ -172,38 +173,39 @@ Let's see the propose flow flowchart A{"IS -layout OR CNB_LAYOUT_DIR - defined?"} -->|Yes| B + defined?"} -->|Yes| BB A -->|No| END + BB["root = "] + BB --> B B{"IS -launch-cache defined?"} -->|Yes|D - B -->|No| E - E{"DOES - layout-dir/image - exist?"} --> |Yes| L - L[...] - E --> |No| M - M[Create layout-dir/image directory] --> O[export-dir = layout-dir/image] - O --> I - D[/Warn: will export to launch cache dir/] --> F + B -->|No| GG + D[/Warn: will export to launch cache dir/] --> FF + FF["root = $launch-cache"] --> GG + GG["export-dir = $root/$layout-dir/$image"] --> F F{"DOES - launch-cache/image - dir exist?"} -->|Yes| G - G[ ...] + $export-dir + exist?"} -->|Yes| P + P["Replace image at $export-dir ‡"] + P --> J F -->|No| H - H[Create launch-cache/image directory] --> N[export-dir = launch-cache/image] - N --> I[Write image to $export-dir in OCI format **] + H[Create $export-dir directory] --> I["Write image to $export-dir †"] I --> J[Calculate manifest's digest] J --> K[/Write digest into report.toml/] K -->END((End)) ``` -Notes: +Notes **†**: - WHEN `-launch-cache` flow is executed - The content of `blobs//` MAY contain symbolic links to content saved in the launch cache to avoid duplicating files. - The content of `blobs//` MAY reference tar files in **uncompressed** format because that's how they are saved in the cache - WHEN `-launch-cache` IS NOT defined - The content of `blobs//` MAY be saved in **compressed** format +Notes **‡**: + - WHEN `the image exists in the file system` + - The idea is to use [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage) internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exits. It means the `blobs` folder MAY contain the blobs directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. + - `Platforms` could include a flag to clean the directory if the user desires it #### `report.toml` (TOML) From 95dc51789144826f0cc08548a1ba07eefb9435a5 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 28 Apr 2022 16:37:27 -0500 Subject: [PATCH 050/222] Some minor fixes Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 6e0bd8d41..ab88f5a82 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -39,7 +39,7 @@ This feature will help to unblock uses cases like Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to add the capability to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format when the new flag `-layout` or the default environment variable `CNB_LAYOUT_DIR` is set. -Let's see some examples for the propose behavior +Let's see some examples of the proposed behavior ## Examples @@ -152,9 +152,9 @@ cd oci # How it Works [how-it-works]: #how-it-works -The lifecycle phases affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) +The lifecycle phase affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) -At high level view the propose solution can be summarize with the following container diagram from the C4 model +At high level view the proposed solution can be summarize with the following container diagram from the C4 model ![](https://i.imgur.com/0OLSK8o.png) @@ -204,8 +204,8 @@ Notes **†**: Notes **‡**: - WHEN `the image exists in the file system` - - The idea is to use [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage) internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exits. It means the `blobs` folder MAY contain the blobs directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. - - `Platforms` could include a flag to clean the directory if the user desires it + - The idea is to use the method [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage). Internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exits. It means the `blobs` directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. + - `Platforms` could include a flag to clean the directory if the user desires it #### `report.toml` (TOML) @@ -225,13 +225,13 @@ Where: # Migration [migration]: #migration - +- No breaking changes were identified + # Drawbacks [drawbacks]: #drawbacks -- We could increase the disk space if we not managed the duplication of saving the layers on disk. Currently the Cache implementation (used when daemon is ON) saved the layers tarballs on disk, because the current proposal is exporting the whole image on disk it will also require more space to save the layers for the OCI format in the `blobs` folder. +- We could increase the disk space if we do not managed the duplication of saving the layers on disk. Currently the Cache implementation (used when daemon is ON) saved the layers tarballs on disk, the proposal is to references those layers in the image exporting on disk to avoid duplication. + # Alternatives [alternatives]: #alternatives @@ -249,6 +249,8 @@ Discuss prior art, both the good and bad. # Unresolved Questions [unresolved-questions]: #unresolved-questions +- Should be this change included on a previous RFC to handle multiple export targets int the Lifecycle? + END((End)) ``` +Notes **‡**: + - WHEN `the image exists in the file system` + - The idea is to use the method [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage). Internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exists. It means the `blobs` directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. + - `Platforms` could include a flag to clean the directory if the user desires it + Notes **†**: - WHEN `-launch-cache` flow is executed - The content of `blobs//` MAY contain symbolic links to content saved in the launch cache to avoid duplicating files. @@ -202,11 +207,6 @@ Notes **†**: - WHEN `-launch-cache` IS NOT defined - The content of `blobs//` MAY be saved in **compressed** format -Notes **‡**: - - WHEN `the image exists in the file system` - - The idea is to use the method [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage). Internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exits. It means the `blobs` directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. - - `Platforms` could include a flag to clean the directory if the user desires it - #### `report.toml` (TOML) The new information to be added into the `report.toml` file can be summarize as follows: From 460fae5cfd3bc1c96cf22248126ae34bf8806d14 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Jun 2022 16:43:02 -0500 Subject: [PATCH 053/222] Updating the latest changes according to the latest PoC Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 240 +++++++++++++++++++------------------ 1 file changed, 122 insertions(+), 118 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index c8e09b317..712d4f845 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -1,6 +1,6 @@ # Meta [meta]: #meta -- Name: Export to OCI format when daemon is enabled +- Name: Export to OCI format - Start Date: 2022-02-22 - Author(s): Juan Bustamante (@jbustamante) - Status: Draft @@ -12,7 +12,7 @@ # Summary [summary]: #summary -When the `Exporter` phase is invoked besides writing into the Daemon or a Registry add the capability (enable explicitly by the user) to save the image to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +Add the capability to the `Exporter` phase to save the image to disk in [OCI Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. # Definitions [definitions]: #definitions @@ -28,16 +28,23 @@ When the `Exporter` phase is invoked besides writing into the Daemon or a Regist # Motivation [motivation]: #motivation -Implementing this new feature will help us to solve the problem of loosing information when the image is saved into the Daemon keeping the image on disk along with the metadata it can be used as input for other tools to offer more capabilities to the end users. - -This feature will help to unblock uses cases like -- OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) -- Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) +Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) will enable new user's workflows in the top of this functionality. For example: + - Reduce the Lifecycle complexity removing the interaction with the Daemon. + - Solve the problem of loosing information when the image is saved into the Daemon keeping the image on disk along with the metadata generated by the Lifecycle. The OCI Image can be used as input for other tools to offer more capabilities to the end users. + - This feature will help to unblock uses cases like + - OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) + - Cosign integration. See [RFC](https://github.com/buildpacks/rfcs/pull/195) # What it is [what-it-is]: #what-it-is -Currently the *Exporter* writes either in an OCI image registry or a docker daemon, the idea is to add the capability to write into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format when the new flag `-layout` or the default environment variable `CNB_LAYOUT_DIR` is set. +The target persona affected by this change are: + +- Platform implementors: because they want to offer this feature, they will have to take care of the responsibility of: + - Pull the require dependencies (runtime image for example) and pass it through the lifecycle + - Save the resulting image to the daemon + +The general idea is to produce an [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) and save it in a file system accesible from the lifecycle execution. Let's see some examples of the proposed behavior @@ -50,162 +57,159 @@ For each case, I will present two ways of invoking the new feature: For both ways the expected output is the same -### Exporting to Daemon with launch cache enabled +### Analyzing run-image from OCI layout format + +Let's suppose the `` or `CNB_OCI_PATH` environment variable points to a directory that contains the `run-image` with the following format ```=shell -> export CNB_LAYOUT=oci -> /cnb/lifecycle/exporter -daemon -launch-cache /launch-cache my-app-image +oci +├── cnbs +   └── sample-stack-run +   ├── blobs +   │   └── sha256 +   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a +   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e +   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a +   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa +   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 +   ├── index.json +   └── oci-layout ``` -Or +And the exporter is invoked as follows ```=shell -> /cnb/lifecycle/exporter -daemon -launch-cache /launch-cache -layout /oci my-app-image +> export CNB_USE_OCI=true +> /cnb/lifecycle/analyzer -run-image cnbs/sample-stack-run:bionic my-app-image ``` -The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/launch-cache/oci/my-app-image/` folder. +Or ```=shell -> cd /launch-cache -> tree . -. -└── launch-cache/ - ├── committed/ - │ ├── io.buildpacks.lifecycle.cache.metadata - │ ├── sha256:65d9067f915e01...tar - │ ├── sha256:6905011516dcf4...tar - │ └── sha256:83d85471d9f8a3...tar - ├── staging - └── oci/ - └── my-app-image/ - ├── blobs/ - │ └── sha256/ - │ ├── 65d9067f915e01...tar -> /launch-cache/committed/sha256:65d9067f915e01...tar - │ ├── 6905011516dcf4...tar -> /launch-cache/committed/sha256:6905011516dcf4...tar - │ └── 83d85471d9f8a3...tar -> /launch-cache/committed/sha256:83d85471d9f8a3...tar - ├── index.json - └── oci-layout - +> /cnb/lifecycle/analyzer -oci -run-image cnbs/sample-stack-run:bionic my-app-image ``` -### Exporting to Daemon without launch cache enabled +The expected output is the analysis metadata [analyzed.toml](https://github.com/buildpacks/spec/blob/main/platform.md#analyzedtoml-toml) with a content similar to this one: -```=shell -> export CNB_LAYOUT=oci -> /cnb/lifecycle/exporter -daemon my-app-image +```=TOML +[run-image] + reference = "sha256:eed6d69be53111ad1d7d3f5d1c037350e6807986feb479a67f36b15f9205a56d" ``` -Or +Where the [ImageID](https://github.com/opencontainers/image-spec/blob/main/config.md#imageid) is calculated according to the OCI specification -```=shell -> /cnb/lifecycle/exporter -daemon -layout oci my-app-image -``` -The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/oci/` folder +### Exporting to OCI layout format + +Similar to the previous example, let's suppose the `` or `CNB_OCI_PATH` environment variable points to a directory that contains the `run-image` with the following format ```=shell -cd oci -> tree . -. -└── oci/ - └── my-app-image/ - ├── blobs/ - │ └── sha256/ - │ ├── 65d9067f915e01...tar - │ ├── 6905011516dcf4...tar - │ └── 83d85471d9f8a3...tar - ├── index.json - └── oci-layout +oci +├── cnbs +   └── sample-stack-run +   ├── blobs +   │   └── sha256 +   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a +   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e +   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a +   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa +   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 +   ├── index.json +   └── oci-layout ``` -### Exporting to a Registry +And the exporter is invoked as follows ```=shell > export CNB_LAYOUT=oci -> /cnb/lifecycle/exporter gcr.io/my-repo/my-app-image +> /cnb/lifecycle/exporter -run-image cnbs/sample-stack-run:bionic my-app-image ``` Or ```=shell -> /cnb/lifecycle/exporter -layout oci gcr.io/my-repo/my-app-image +> /cnb/lifecycle/exporter -oci -run-image cnbs/sample-stack-run:bionic my-app-image ``` -The expected output is the `my-app-image` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format into the `/oci/` folder +The expected output is the application `` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format ```=shell -cd oci -> tree . -. -└── oci/ - └── my-app-image/ - ├── blobs/ - │ └── sha256/ - │ ├── 65d9067f915e01...tar - │ ├── 6905011516dcf4...tar - │ └── 83d85471d9f8a3...tar - ├── index.json - └── oci-layout +├── cnbs +│   └── sample-stack-run +│   ├── blobs +│   │   └── sha256 +│   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a +│   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e +│   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a +│   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa +│   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 +│   ├── index.json +│   └── oci-layout +└── my-app-image + ├── blobs + │   └── sha256 + │   ├── 14eaea7168b1fc4b8b30f7a20f7609335cc3dbcfb6d4c1afeb1e5daefd26cdf9 + │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e -> ../../../cnbs/sample-stack-run/blobs/sha256/219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e + │   ├── 410ce2030625414163d565a56bddc6587dbc49fa2a815af5e26bb08968dec7d2 + │   ├── 54890b0245f8b234be46817f731b7b981d9ee2ea8d5a76380d91bb9abb001cdb + │   ├── 58b81a67d77e732ef07c4b995b6a7e099e5aa772d1805da7cf929d80a3fa044e + │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa -> ../../../cnbs/sample-stack-run/blobs/sha256/63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa + │   ├── 68224c8aa806b9ec8cecf2282b7b10cdf1c615785652bcc85f3a79bd06e60384 + │   ├── 6977801a3b8ac0c4a76ea22fa4dc0541fe8fc6ba964883ee0a21b5736f8acee9 + │   ├── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 -> ../../../cnbs/sample-stack-run/blobs/sha256/6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 + │   ├── 7c83d7d24ac43dbdd419d15abb849b6d155d0bc361eca0908a8ae5aefcd55557 + │   └── addd2357f3a0175410ab8f9303e747cc72af9aaeb0e7402d3fd3f144adb29db5 + ├── index.json + └── oci-layout +``` + +Also, we are proposing to use symbolic links to reference the blobs from the `run-image` this will help to save some hard drive space for the end user. + +### Using -oci flag in combination with --daemon or --publish flags + +Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: +```=shell +> /cnb/lifecycle/exporter -oci -daemon -run-image cnbs/sample-stack-run:bionic my-app-image + +ERROR: exporting to multiples target is not allowed ``` # How it Works [how-it-works]: #how-it-works -The lifecycle phase affected by this new behavior is: [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) +Before defining the detail behavior, lets remember some required terminology + +**Terminology** -At high level view the proposed solution can be summarize with the following container diagram from the C4 model +- An **image reference** refers to either a tag reference or digest reference. +- A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. -![](https://i.imgur.com/0OLSK8o.png) +The lifecycle phases affected by this new behavior are: + - [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) + - [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) + - [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/) +At high level view the proposed solution can be summarize with the following system landscape diagram from the C4 model + +![](https://i.imgur.com/y972lTD.png) Notice that we are relying on the OCI format Specification to expose the data for `Platforms` -The following new input is proposed to be added to this phase - -| Input | Environment Variable | Default Value | Description -|-------------------|-----------------------|--------------------------|---------------------- -| `` | `CNB_LAYOUT_DIR` | "" | The root directory where the OCI image will be written. The presence of a none empty value for this environment variable will enable the feature. | - -Let's see the proposed flow - -```mermaid -flowchart - A{"IS -layout OR - CNB_LAYOUT_DIR - defined?"} -->|Yes| BB - A -->|No| END - BB["root = "] - BB --> B - B{"IS -launch-cache - defined?"} -->|Yes|D - B -->|No| GG - D[/Warn: will export to launch cache dir/] --> FF - FF["root = $launch-cache"] --> GG - GG["export-dir = $root/$layout-dir/$image"] --> F - F{"DOES - $export-dir - exist?"} -->|Yes| P - P["Replace image at $export-dir ‡"] - P --> J - F -->|No| H - H[Create $export-dir directory] --> I["Write image to $export-dir †"] - I --> J[Calculate manifest's digest] - J --> K[/Write digest into report.toml/] - K -->END((End)) -``` +The following new input are proposed to be added to these phases -Notes **‡**: - - WHEN `the image exists in the file system` - - The idea is to use the method [ReplaceImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.ReplaceImage). Internally this method uses [WriteImage](https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/layout#Path.WriteImage) which will skip to write blobs that already exists. It means the `blobs` directory MAY contain blobs which are not referenced by any of the refs, which is valid according to the OCI image specification. - - `Platforms` could include a flag to clean the directory if the user desires it + | Input | Environment Variable | Default Value | Description + |-------------------|-----------------------|--------------------------|---------------------- + | `` | `CNB_USE_OCI` | false | Analyze or Export image from/to OCI layout format on disk | + | `` | `CNB_OCI_PATH` | /oci | Path to oci directory where the images are saved | -Notes **†**: - - WHEN `-launch-cache` flow is executed - - The content of `blobs//` MAY contain symbolic links to content saved in the launch cache to avoid duplicating files. - - The content of `blobs//` MAY reference tar files in **uncompressed** format because that's how they are saved in the cache - - WHEN `-launch-cache` IS NOT defined - - The content of `blobs//` MAY be saved in **compressed** format +- WHEN `the new flag -oci or the default environment variable CNB_USE_OCI are set to true` during the phase invocation THEN the feature will be enabled. +The image look up will be done following this rules: + - WHEN `the image points to a tag reference` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - WHEN `the image points to a digest reference` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` #### `report.toml` (TOML) @@ -218,7 +222,7 @@ digest = "" manifest-size = "" ``` Where: -- **If** the app image was exported using the `-layer` flag, the export section will be added to the report +- **If** the app image was exported to OCI format on disk, the export section will be added to the report - `digest` MUST contain the image digest calculated based on compressed layers - `manifest-size` MUST contain the manifest size in bytes From 580c8ba85b5ab67a9bf5bade7198c16f945fc8cc Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Jun 2022 16:54:56 -0500 Subject: [PATCH 054/222] Formatting changes Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 712d4f845..1f4bc4d9c 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -19,10 +19,12 @@ Add the capability to the `Exporter` phase to save the image to disk in [OCI Lay - A [Platform](https://buildpacks.io/docs/concepts/components/platform/) uses a lifecycle, Buildpacks (packaged in a builder), and application source code to produce an OCI image. - A [Lifecycle](https://buildpacks.io/docs/concepts/components/lifecycle/) orchestrates Buildpacks execution, then assembles the resulting artifacts into a final app image. -- A Daemon is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. -- A Registry is a long-running service used for storing and retrieving container images. -- A digest reference refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form /@ which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. -- A Image Manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. +- A **Daemon** is a service, popularized by Docker, for downloading container images, and executing and managing containers from those images. +- A **Registry** is a long-running service used for storing and retrieving container images. +- An **image reference** refers to either a tag reference or digest reference. +- A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. +- A **image Manifest** provides a configuration and set of layers for a single container image for a specific architecture and operating system. - An [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) is the directory structure for OCI content-addressable blobs and location-addressable references. # Motivation @@ -40,11 +42,10 @@ Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blo The target persona affected by this change are: -- Platform implementors: because they want to offer this feature, they will have to take care of the responsibility of: +- **Platform implementors**: they will have to take care of the responsibility of: - Pull the require dependencies (runtime image for example) and pass it through the lifecycle - - Save the resulting image to the daemon -The general idea is to produce an [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) and save it in a file system accesible from the lifecycle execution. +The general idea is to produce an [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) and save it in a file system accesible from the lifecycle execution. Let's see some examples of the proposed behavior @@ -178,14 +179,6 @@ ERROR: exporting to multiples target is not allowed # How it Works [how-it-works]: #how-it-works -Before defining the detail behavior, lets remember some required terminology - -**Terminology** - -- An **image reference** refers to either a tag reference or digest reference. -- A **tag reference** refers to an identifier of form `/:` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. -- A **digest reference** refers to a [content addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) identifier of form `/@` which locates an image manifest in an [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/master/spec.md) compliant registry. - The lifecycle phases affected by this new behavior are: - [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) - [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) From d7f4ec80bd6c937b5eb4db0faa6a56a4944f1d1c Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Jun 2022 17:12:44 -0500 Subject: [PATCH 055/222] Some other changes Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 1f4bc4d9c..28b59c83c 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -45,7 +45,7 @@ The target persona affected by this change are: - **Platform implementors**: they will have to take care of the responsibility of: - Pull the require dependencies (runtime image for example) and pass it through the lifecycle -The general idea is to produce an [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) and save it in a file system accesible from the lifecycle execution. +The general idea is to produce an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and save it in a file system accesible from the lifecycle execution. Let's see some examples of the proposed behavior @@ -77,7 +77,7 @@ oci    └── oci-layout ``` -And the exporter is invoked as follows +And the analyzer is invoked as follows ```=shell > export CNB_USE_OCI=true @@ -203,10 +203,11 @@ The image look up will be done following this rules: - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - WHEN `the image points to a digest reference` - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - A part from the look up, the logic for each phase should remain the same #### `report.toml` (TOML) -The new information to be added into the `report.toml` file can be summarize as follows: +The new information to be added into the `report.toml` file can be summarize as follows: ```toml [export] @@ -227,27 +228,24 @@ Where: # Drawbacks [drawbacks]: #drawbacks -- We could increase the disk space if we do not managed the duplication of saving the layers on disk. Currently the Cache implementation (used when daemon is ON) saved the layers tarballs on disk, the proposal is to references those layers in the image exporting on disk to avoid duplication. - +- We could increase the disk space if we do not managed the duplication of saving the layers on disk. The proposal suggest to use symbolic links to reference layers on disk and avoid duplication. # Alternatives [alternatives]: #alternatives - - +- What other designs have been considered? + - Doing nothing, just keep exporting only to Daemon or Registry +- Why is this proposal the best? [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is an standard from which other tools can create a [OCI Runtime Specification bundle](https://github.com/opencontainers/runtime-spec/blob/v1.0.0/bundle.md) exporting to this format enables Platforms to implement any feature in the top of this format +- What is the impact of not doing this? Probably will never remove the Daemon support in the Lifecycle # Prior Art [prior-art]: #prior-art -Discuss prior art, both the good and bad. +- Discussion around removing the Daemon support [RFC](https://github.com/buildpacks/rfcs/blob/jjbustamante/feature/deprecate-daemon/text/0000-deprecate-daemon.md) # Unresolved Questions [unresolved-questions]: #unresolved-questions -- Should this change be included on a previous RFC to handle multiple export targets int the Lifecycle? -- Currently the *Launch Cache* saves uncompressed tarballs. Is this by design? Is there any reason for those tarballs to do not be saved compressed? -Let's suppose the `` or `CNB_OCI_PATH` environment variable points to a directory that contains the `run-image` with the following format +Let's suppose a directory exits and it has the following structure: ```=shell oci -├── cnbs -   └── sample-stack-run -   ├── blobs -   │   └── sha256 -   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a -   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e -   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a -   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa -   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 -   ├── index.json -   └── oci-layout +├── cnb +│ └── my-stack-run +│ ├── blobs +│ │ └── sha256 +│ │ └── 1bcd..x +│ ├── index.json +│ └── oci-layout +├── foo +│ └── my-cache +│ ├── blobs +│ │ └── sha256 +│ │ └── 1f591..a +│ ├── index.json +│ └── oci-layout +└── bar + └── my-previous-app + ├── blobs + │ └── sha256 + │ └── 1efg..w + ├── index.json + └── oci-layout ``` -And the analyzer is invoked as follows +For each case, I will present two ways of enabling the new capability: + +- Using an environment Variable +- Using a new `oci` flag + +In any case the expected output is the same. + +#### Analyze phase + +##### Analyzing run-image ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/analyzer -run-image cnbs/sample-stack-run:bionic my-app-image -``` +> /cnb/lifecycle/analyzer -run-image cnb/my-stack-run:bionic my-app-image -Or +# OR -```=shell -> /cnb/lifecycle/analyzer -oci -run-image cnbs/sample-stack-run:bionic my-app-image -``` +> /cnb/lifecycle/analyzer -oci -run-image cnb/my-stack-run:bionic my-app-image -The expected output is the analysis metadata [analyzed.toml](https://github.com/buildpacks/spec/blob/main/platform.md#analyzedtoml-toml) with a content similar to this one: +# expected output +# analyzed.toml file with the usual `run-image.reference` -```=TOML -[run-image] - reference = "sha256:eed6d69be53111ad1d7d3f5d1c037350e6807986feb479a67f36b15f9205a56d" ``` -Where the [ImageID](https://github.com/opencontainers/image-spec/blob/main/config.md#imageid) is calculated according to the OCI specification +##### Analyzing previous-image + ```=shell +> export CNB_USE_OCI=true +> /cnb/lifecycle/analyzer -run-image cnb/my-stack-run:bionic -previous-image bar/my-previous-app my-app-image + +# OR -### Exporting to OCI layout format +> /cnb/lifecycle/analyzer -oci -run-image cnb/my-stack-run:bionic-previous-image bar/my-previous-app my-app-image -Similar to the previous example, let's suppose the `` or `CNB_OCI_PATH` environment variable points to a directory that contains the `run-image` with the following format +# expected output +# analyzed.toml file with the usual `run-image.reference` and `previos-image.reference` +``` + + +##### Analyzing run-image not saved on disk ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/exporter -run-image cnbs/sample-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image + +# OR + +> /cnb/lifecycle/analyzer -oci -run-image cnb/bad-run-image my-app-image + +# expected output + +ERROR: the run-image could not be found at path: /oci/cnb/bad-run-image ``` -Or +##### Analyzing without run-image argument ```=shell -> /cnb/lifecycle/exporter -oci -run-image cnbs/sample-stack-run:bionic my-app-image +> export CNB_USE_OCI=true +> /cnb/lifecycle/analyzer my-app-image + +# OR + +> /cnb/lifecycle/analyzer -oci my-app-image + +# expected output + +ERROR: -run-image is required when OCI feature is enabled ``` -The expected output is the application `` exported in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format +#### Export phase + +##### Export to OCI + ```=shell +> export CNB_USE_OCI=true +> /cnb/lifecycle/exporter -run-image cnb/my-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/exporter -oci -run-image cnb/my-stack-run:bionic my-app-image + +# expected output +# the store folder will be updated with the application image as follows + oci -├── cnbs - │   └── sample-stack-run - │   ├── blobs - │   │   └── sha256 - │   │   ├── 1f59171fcf9a8c2a78192d4dfbf88e6f0258e24f21798ffe015c07682d3a944a - │   │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e - │   │   ├── 63882e64890d1adea5824dbb9bc6a812140d6b85752bf74a0a26fca78e1baf5a - │   │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa - │   │   └── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 - │   ├── index.json - │   └── oci-layout - └── my-app-image - ├── blobs - │   └── sha256 - │   ├── 14eaea7168b1fc4b8b30f7a20f7609335cc3dbcfb6d4c1afeb1e5daefd26cdf9 - │   ├── 219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e -> ../../../cnbs/sample-stack-run/blobs/sha256/219480c73c20efd82d425207e7555eba09cb113c845250cef09ba376e1dd506e - │   ├── 410ce2030625414163d565a56bddc6587dbc49fa2a815af5e26bb08968dec7d2 - │   ├── 54890b0245f8b234be46817f731b7b981d9ee2ea8d5a76380d91bb9abb001cdb - │   ├── 58b81a67d77e732ef07c4b995b6a7e099e5aa772d1805da7cf929d80a3fa044e - │   ├── 63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa -> ../../../cnbs/sample-stack-run/blobs/sha256/63cb781c9d22ed765584437fea6db568f79be248ce40fc6695017d3ef3e8caaa - │   ├── 68224c8aa806b9ec8cecf2282b7b10cdf1c615785652bcc85f3a79bd06e60384 - │   ├── 6977801a3b8ac0c4a76ea22fa4dc0541fe8fc6ba964883ee0a21b5736f8acee9 - │   ├── 6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 -> ../../../cnbs/sample-stack-run/blobs/sha256/6b534265b45a4c09a5e05dde5ccf2450ea505481658a1c9ae9e0aae32362e941 - │   ├── 7c83d7d24ac43dbdd419d15abb849b6d155d0bc361eca0908a8ae5aefcd55557 - │   └── addd2357f3a0175410ab8f9303e747cc72af9aaeb0e7402d3fd3f144adb29db5 - ├── index.json - └── oci-layout -``` +├── cnb +│ └── my-stack-run +│ ├── blobs +│ │ └── sha256 +│ │ └── 1bcd..x +│ ├── index.json +│ └── oci-layout +├── foo +│ └── // omiting for simplicity +├── bar +│ └── // omiting for simplicity +└── my-app-image + └── blobs + ├── sha256 + │ ├── 1bcd..x + │ ├── 2f789..d + │ └── 3g234..f + ├── index.json + └── oci-layout -Also, we are proposing to use symbolic links to reference the blobs from the `run-image` as this will help to save some hard drive space for the end user. +``` -### Using -oci flag in combination with --daemon or --publish flags +##### Using -oci flag in combination with --daemon or --publish flags Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: ```=shell -> /cnb/lifecycle/exporter -oci -daemon -run-image cnbs/sample-stack-run:bionic my-app-image +> export CNB_USE_OCI=true +> /cnb/lifecycle/exporter -daemon -run-image cnb/my-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/exporter -oci -daemon -run-image cnb/my-stack-run:bionic my-app-image ERROR: exporting to multiples target is not allowed ``` @@ -197,8 +264,8 @@ Notice that we are relying on the OCI format Specification to expose the data fo The following new inputs are proposed to be added to these phases - | Input | Environment Variable | Default Value | Description - |-------------------|-----------------------|--------------------------|---------------------- + | Input | Environment Variable | Default Value | Description + |-------|-----------------------|---------------|-------------- | `` | `CNB_USE_OCI` | false | Analyze or Export image from/to OCI layout format on disk | | `` | `CNB_OCI_PATH` | /oci | Path to oci directory where the images are saved | From f9e225f8108b8496a6a77d81fd079a73ef3ab7e7 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 31 Oct 2022 12:09:03 -0500 Subject: [PATCH 081/222] removing the plan for daemon removal Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index a4ac5cc82..6fb8a9346 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -2,7 +2,7 @@ [meta]: #meta - Name: Export to OCI format - Start Date: 2022-02-22 -- Author(s): Juan Bustamante (@jbustamante) +- Author(s): Juan Bustamante (@jjbustamante) - Status: Draft - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) @@ -277,6 +277,9 @@ The image look up will be done following these rules: - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - Apart from the look up, the logic for each phase should remain the same + + + ## Proof of concept In order to validate the feasibility of the proposed feature, we developed a proof of concept with one of the most important side effects this capability can add into the project: **Removing the Daemon Support**. You can also check a recording with the demo in the following [link](https://drive.google.com/file/d/1W1125OHuyUlx88BRroUTLBfrFHhFM5A9/view?usp=sharing) @@ -334,23 +337,6 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - No breaking changes were identified -## For removing Daemon support in near future - -I propose the following high level strategy to accomplish the goal - -- Approve this RFC and add the export to OCI layout format feature in Lifecycle -- Implement in Pack a new workflow when the `-daemon` flag is enabled based on what we did with the PoC. The workflow could be enabled by the user in **experimental mode** - - Make noise in the community so people can help us using it - - Collect feedback about it - - Correct issues until we feel confortable -- Turn off the actual `-daemon` implementation and replace it with the `-oci` by default in Pack - - Maybe we can also add a new flag to **re-use the old implementation** in case the user needs it - - Keep doing noise in the community - - Collect feedback about it - - Correct issues until we feel confortable -- Schedule the removal of the Daemon support in Lifecycle version X.Y.Z. -- Remove the actual daemon support from the code base for Lifecycle version X.Y.Z - # Drawbacks [drawbacks]: #drawbacks From f79bb237770c38fbaeec5a6915ffa13787c6b691 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 3 Nov 2022 09:46:41 -0500 Subject: [PATCH 082/222] adding the possibility to allow partial images on disk Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 6fb8a9346..7c922ac74 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -43,7 +43,7 @@ The current process, executed by the lifecycle, does not take into consideration ### What use cases does it support? Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) will enable new user's workflows on top of this functionality. For example: - - Reduce the Lifecycle complexity removing the interaction with the Daemon. + - It provides a mechanism to reduce the Lifecycle complexity by removing the interaction with the Daemon in the future. - Solve the problem of losing information when the image is saved into the Daemon, keeping the image on disk along with the metadata generated by the Lifecycle. The OCI Image can be used as input for other tools to offer more capabilities to the end users. - This feature will help to unblock uses cases like - OCI annotations. See [RFC](https://github.com/buildpacks/rfcs/pull/196) @@ -52,7 +52,11 @@ Exporting to [OCI Image Layout](https://github.com/opencontainers/image-spec/blo ### What is the expected outcome? -Lifecycle will be capable of exporting application image into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +Lifecycle will be capable of exporting the application image into disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. The image saved on disk could have the following considerations: + +- The `blobs` directory MAY be missing `base image` or `run image` blobs. These layers may not be needed on disk as they could be already accessible in a blob store. +- The `blobs` directory SHOULD always have buildpacks generated `blobs`. + # What it is [what-it-is]: #what-it-is @@ -63,6 +67,8 @@ The target personas affected by this change are: - **Platform implementors**: they will have to take care of the responsibility of creating a store resource on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and pass it through the lifecycle during the phases execution. +The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations, based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs COULD be those that are already available in a daemon or registry. + Let's see some examples of the proposed behavior ## Examples From 966488a11a5690c8112f6794ac2a8c6caa34a556 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 3 Nov 2022 18:19:29 -0500 Subject: [PATCH 083/222] extending the examples section Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 256 +++++++++++++++++++++++++++++++------ 1 file changed, 215 insertions(+), 41 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 7c922ac74..be61b28dc 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -67,7 +67,7 @@ The target personas affected by this change are: - **Platform implementors**: they will have to take care of the responsibility of creating a store resource on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and pass it through the lifecycle during the phases execution. -The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations, based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs COULD be those that are already available in a daemon or registry. +The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations, based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs SHOULD be those that are already available in a daemon or registry. Let's see some examples of the proposed behavior @@ -92,10 +92,21 @@ Let's suppose a directory exits and it has the following structure: ```=shell oci ├── cnb -│ └── my-stack-run +│ ├── my-full-stack-run +│ │ ├── blobs +│ │ │ └── sha256 +│ │ │ ├── 1f59...944a // manifest +│ │ │ ├── 6388...af5a // config +│ │ │ ├── 824b...f984e +│ │ │ ├── f5f7...5b38 +│ │ │ └── 870e...f1b09 +│ │ ├── index.json +│ │ └── oci-layout +│ └── my-partial-stack-run │ ├── blobs │ │ └── sha256 -│ │ └── 1bcd..x +│ │ ├── 1f59...944a // manifest +│ │ └── 6388...af5a // config │ ├── index.json │ └── oci-layout ├── foo @@ -105,16 +116,29 @@ oci │ │ └── 1f591..a │ ├── index.json │ └── oci-layout -└── bar - └── my-previous-app - ├── blobs - │ └── sha256 - │ └── 1efg..w +├── bar +│ └── my-previous-app +│ ├── blobs +│ │ └── sha256 +│ │ └── 1efg..w +│ ├── index.json +│ └── oci-layout +└── my-app-image + └── blobs + ├── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer ├── index.json └── oci-layout ``` -For each case, I will present two ways of enabling the new capability: +The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** represents the same image but the partial one has missing `blobs`, those `blobs` are the layers that are already available in the store from it came from. + +For each example case, I will present two ways of enabling the new capability: - Using an environment Variable - Using a new `oci` flag @@ -123,52 +147,51 @@ In any case the expected output is the same. #### Analyze phase -##### Analyzing run-image +##### Analyzing run-image full saved on disk ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/analyzer -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -oci -run-image cnb/my-full-stack-run:bionic my-app-image # expected output # analyzed.toml file with the usual `run-image.reference` ``` -##### Analyzing previous-image +##### Analyzing run-image partial saved on disk - ```=shell +```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/analyzer -run-image cnb/my-stack-run:bionic -previous-image bar/my-previous-app my-app-image +> /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/my-stack-run:bionic-previous-image bar/my-previous-app my-app-image +> /cnb/lifecycle/analyzer -oci -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # expected output -# analyzed.toml file with the usual `run-image.reference` and `previos-image.reference` +# analyzed.toml file with the usual `run-image.reference` +# there should be no change behavior in the phase even thought the image has missing blobs + ``` - - ##### Analyzing run-image not saved on disk ```=shell @@ -201,54 +224,96 @@ ERROR: -run-image is required when OCI feature is enabled #### Export phase -##### Export to OCI +##### Export to OCI using run-image full saved on disk +```=shell +> export CNB_USE_OCI=true +> /cnb/lifecycle/exporter -run-image cnb/my-full-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/exporter -oci -run-image cnb/my-full-stack-run:bionic my-app-image + +# expected output +# the store folder will be updated with the application image as follows + +oci +├── cnb +│ └── my-full-stack-run +│ ├── blobs +│ │ └── sha256 +│ │ ├── 1f59...944a // manifest +│ │ ├── 6388...af5a // config +│ │ ├── 824b...f984e +│ │ ├── f5f7...5b38 +│ │ └── 870e...f1b09 +│ ├── index.json +│ └── oci-layout +├── // some folders omitted for simplicity +└── my-app-image + └── blobs + ├── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout + +``` + +As we can see, the application image `my-app-image` contains a **full** copy of the layers in its `blobs` folder. + + +##### Export to OCI using run-image partially saved on disk ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/exporter -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -run-image cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -oci -run-image cnb/my-partial-stack-run:bionic my-app-image # expected output # the store folder will be updated with the application image as follows oci ├── cnb -│ └── my-stack-run +│ └── my-partial-stack-run │ ├── blobs │ │ └── sha256 -│ │ └── 1bcd..x +│ │ ├── 1f59...944a // manifest +│ │ ├── 6388...af5a // config │ ├── index.json │ └── oci-layout -├── foo -│ └── // omiting for simplicity -├── bar -│ └── // omiting for simplicity +├── // some folders omitted for simplicity └── my-app-image └── blobs ├── sha256 - │ ├── 1bcd..x - │ ├── 2f789..d - │ └── 3g234..f + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ └── 3g234..f // buildpack layer ├── index.json └── oci-layout ``` +As we can see, the application image `my-app-image` has missing `blobs` because they were not provided as input and the lifecycle just **skip writing** those layers on disk. + ##### Using -oci flag in combination with --daemon or --publish flags Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: ```=shell > export CNB_USE_OCI=true -> /cnb/lifecycle/exporter -daemon -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -daemon -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -daemon -run-image cnb/my-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -oci -daemon -run-image cnb/my-full-stack-run:bionic my-app-image ERROR: exporting to multiples target is not allowed ``` @@ -281,10 +346,119 @@ The image look up will be done following these rules: - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - WHEN `the image points to a digest reference` - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - - Apart from the look up, the logic for each phase should remain the same + - Apart for the image look up, the logic for each phase should remain the same and the lifecycle will write the layers on disk according to the following rules: + - Any layer generated by a buildpack SHOULD always be written to disk + - Layers from `base-image` or `run-image` MAY be written on disk if they were provided + +Let's review our previous examples. + +## Examples + +In all the examples the new feature is enabled by the use of the new flag `-oci` or by setting the new environment variable `CNB_USE_OCI` to true. + +#### Analyze phase + +##### Analyzing run-image full saved on disk + +Arguments received: + + - `run-image`: `cnb/my-full-stack-run:bionic` + - `image`: `my-app-image` + +The `` is set with the default value `/oci` + +Lifecycle applies the rules for looking up the images: + - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + - Lifecycle validates the `annotations` map in the **Image Manifest** contains a key-value paired `"org.opencontainers.image.ref.name" : "bionic"` + - In case of the *application image* it will look at path `/oci/my-app-image` + + Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated + +##### Analyzing run-image partial saved on disk + +Arguments received: + + - `run-image`: `cnb/my-full-partial-run:bionic` + - `image`: `my-app-image` + +The `` is set with the default value `/oci` + +Noticed the structure of the `run-image` provided + +```=shell +oci +├── cnb + └── my-partial-stack-run + ├── blobs + │ └── sha256 + │ ├── 1f59...944a // manifest + │ └── 6388...af5a // config + ├── index.json + └── oci-layout +``` + +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/oci/cnb/my-partial-stack-run` and it determines a partial image was provided and execute the phase logic with the information from the **Image Manifest** and the **Image Config** + +##### Analyzing previous-image + +Arguments received: + +- `run-image`: `cnb/my-full-stack-run:bionic` +- `previous-image`: `bar/my-previous-app` +- `image`: `my-app-image` + +The `` is set with the default value `/oci` + +`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/oci/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + +##### Analyzing run-image not saved on disk + +Arguments received: + +- `run-image`: `cnb/bad-run-image` +- `image`: `my-app-image` + +The `` is set with the default value `/oci` + +In this case, Lifecycle will will look at path `/oci/bad-run-image` and because the path doesn't exists then an error must be thrown. + +##### Analyzing without run-image argument + +Arguments received: + +- `image`: `my-app-image` + +The `` is set with the default value `/oci` + +When the feature is enabled, Lifecycle requires any input image must be available on disk, because the `run-image` reference was not provided Lifecycle must thrown an error. + +#### Export phase + +##### Export to OCI using run-image full saved on disk + +Arguments received: + +- `run-image`: `cnb/my-full-stack-run:bionic` +- `image`: `my-app-image` + +The `` is set with the default value `/oci` + +Lifecycle applies the rules for looking up the images: + - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + - Lifecycle validates the `annotations` map in the **Image Manifest** contains a key-value paired `"org.opencontainers.image.ref.name" : "bionic"` + - Lifecycle determines the `run-image` is a full image reference (it contains all the blobs) + - Lifecycle writes the *application image* at path `/oci/my-app-image` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format including the layers from the `run-image` + +##### Export to OCI using run-image partially saved on disk + +Arguments received: +- `run-image`: `cnb/my-partial-stack-run:bionic` +- `image`: `my-app-image` +The `` is set with the default value `/oci` +Lifecycle behaves similar to the previous example, but during the process of writing the output application image to disk it will skip the layers that are missing in the `blobs` folder at path `/oci/cnb/my-partial-stack-run` ## Proof of concept From 9eb99ad29e79ddf2cef7339d085cd3dda4d9dbb9 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 8 Nov 2022 08:23:40 -0500 Subject: [PATCH 084/222] Update text/0000-export-to-oci.md Fixes suggested by Natalie Co-authored-by: Natalie Arellano Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index be61b28dc..e7fab3b00 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -67,7 +67,7 @@ The target personas affected by this change are: - **Platform implementors**: they will have to take care of the responsibility of creating a store resource on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format and pass it through the lifecycle during the phases execution. -The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations, based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs SHOULD be those that are already available in a daemon or registry. +The process of writing any image on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format could be expensive in terms of hard drive space or IO operation (compressing or uncompressing layers). In order to provide flexibility for the implementation, the `analyzer` or `exporter` binaries only require the *Image Manifest* and the *Image Config* to execute their operations on the previous image and run image; based on this, we proposed the Lifecycle can be configured to work with a partial representation of the images on disk, meaning that some blobs MAY be missing (which is ok according to the [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format). The missing blobs SHOULD be those that are already available in a daemon or registry. Let's see some examples of the proposed behavior From e6e01fdf0cb80982718214fee3bd7169ad80b3ea Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 9 Nov 2022 17:16:12 -0500 Subject: [PATCH 085/222] renaming flags to use layout instead of oci Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 64 +++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index e7fab3b00..ba0c4ec61 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -141,7 +141,7 @@ The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** repr For each example case, I will present two ways of enabling the new capability: - Using an environment Variable -- Using a new `oci` flag +- Using a new `-layout` flag In any case the expected output is the same. @@ -150,12 +150,12 @@ In any case the expected output is the same. ##### Analyzing run-image full saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image # expected output # analyzed.toml file with the usual `run-image.reference` @@ -165,12 +165,12 @@ In any case the expected output is the same. ##### Analyzing run-image partial saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # expected output # analyzed.toml file with the usual `run-image.reference` @@ -181,12 +181,12 @@ In any case the expected output is the same. ##### Analyzing previous-image ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image # expected output # analyzed.toml file with the usual `run-image.reference` and `previos-image.reference` @@ -195,12 +195,12 @@ In any case the expected output is the same. ##### Analyzing run-image not saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image # OR -> /cnb/lifecycle/analyzer -oci -run-image cnb/bad-run-image my-app-image +> /cnb/lifecycle/analyzer -layout -run-image cnb/bad-run-image my-app-image # expected output @@ -210,12 +210,12 @@ ERROR: the run-image could not be found at path: /oci/cnb/bad-run-image ##### Analyzing without run-image argument ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/analyzer my-app-image # OR -> /cnb/lifecycle/analyzer -oci my-app-image +> /cnb/lifecycle/analyzer -layout my-app-image # expected output @@ -227,12 +227,12 @@ ERROR: -run-image is required when OCI feature is enabled ##### Export to OCI using run-image full saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/exporter -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout -run-image cnb/my-full-stack-run:bionic my-app-image # expected output # the store folder will be updated with the application image as follows @@ -270,12 +270,12 @@ As we can see, the application image `my-app-image` contains a **full** copy of ##### Export to OCI using run-image partially saved on disk ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/exporter -run-image cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -run-image cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout -run-image cnb/my-partial-stack-run:bionic my-app-image # expected output # the store folder will be updated with the application image as follows @@ -303,17 +303,17 @@ oci As we can see, the application image `my-app-image` has missing `blobs` because they were not provided as input and the lifecycle just **skip writing** those layers on disk. -##### Using -oci flag in combination with --daemon or --publish flags +##### Using -layout flag in combination with --daemon or --publish flags Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: ```=shell -> export CNB_USE_OCI=true +> export CNB_USE_OCI_LAYOUT=true > /cnb/lifecycle/exporter -daemon -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -oci -daemon -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout -daemon -run-image cnb/my-full-stack-run:bionic my-app-image ERROR: exporting to multiples target is not allowed ``` @@ -337,15 +337,15 @@ The following new inputs are proposed to be added to these phases | Input | Environment Variable | Default Value | Description |-------|-----------------------|---------------|-------------- - | `` | `CNB_USE_OCI` | false | Analyze or Export image from/to OCI layout format on disk | - | `` | `CNB_OCI_PATH` | /oci | Path to oci directory where the images are saved | + | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | + | `` | `CNB_OCI_LAYOUT_PATH` | /oci | Path to oci directory where the images are saved | -- WHEN `the new flag -oci or the default environment variable CNB_USE_OCI are set to true` during the phase invocation THEN the feature will be enabled. +- WHEN `the new flag -layout or the default environment variable CNB_USE_OCI_LAYOUT are set to true` during the phase invocation THEN the feature will be enabled. The image look up will be done following these rules: - WHEN `the image points to a tag reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - WHEN `the image points to a digest reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - Apart for the image look up, the logic for each phase should remain the same and the lifecycle will write the layers on disk according to the following rules: - Any layer generated by a buildpack SHOULD always be written to disk - Layers from `base-image` or `run-image` MAY be written on disk if they were provided @@ -354,7 +354,7 @@ Let's review our previous examples. ## Examples -In all the examples the new feature is enabled by the use of the new flag `-oci` or by setting the new environment variable `CNB_USE_OCI` to true. +In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting the new environment variable `CNB_USE_OCI_LAYOUT` to true. #### Analyze phase @@ -365,7 +365,7 @@ Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` Lifecycle applies the rules for looking up the images: - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. @@ -381,7 +381,7 @@ Arguments received: - `run-image`: `cnb/my-full-partial-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` Noticed the structure of the `run-image` provided @@ -407,7 +407,7 @@ Arguments received: - `previous-image`: `bar/my-previous-app` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` `run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/oci/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. @@ -418,7 +418,7 @@ Arguments received: - `run-image`: `cnb/bad-run-image` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` In this case, Lifecycle will will look at path `/oci/bad-run-image` and because the path doesn't exists then an error must be thrown. @@ -428,7 +428,7 @@ Arguments received: - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` When the feature is enabled, Lifecycle requires any input image must be available on disk, because the `run-image` reference was not provided Lifecycle must thrown an error. @@ -441,7 +441,7 @@ Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` Lifecycle applies the rules for looking up the images: - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. @@ -456,7 +456,7 @@ Arguments received: - `run-image`: `cnb/my-partial-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/oci` Lifecycle behaves similar to the previous example, but during the process of writing the output application image to disk it will skip the layers that are missing in the `blobs` folder at path `/oci/cnb/my-partial-stack-run` @@ -465,7 +465,7 @@ Lifecycle behaves similar to the previous example, but during the process of wri In order to validate the feasibility of the proposed feature, we developed a proof of concept with one of the most important side effects this capability can add into the project: **Removing the Daemon Support**. You can also check a recording with the demo in the following [link](https://drive.google.com/file/d/1W1125OHuyUlx88BRroUTLBfrFHhFM5A9/view?usp=sharing) As mentioned earlier, if we want to remove the daemon support in the Lifecycle, then all the responsibility to deal with it goes into the platforms implementors, that means, for example: -- Pull the require dependencies (runtime image for example), save them on disk in OCI layout format and pass it through the lifecycle using the `` parameter +- Pull the require dependencies (runtime image for example), save them on disk in OCI layout format and pass it through the lifecycle using the `` parameter - Push the application image (exported in OCI layout format) into the Daemon, because that is what users are expecting. During the proof of concept implementation I choose to use [skopeo](https://github.com/containers/skopeo) tool to solve the problem of interacting with the Daemon. The reason to do it was **simplicity** for the PoC developed but we believe this is a good subject to talk about with the community. From a38c474d4604851ae92f35538264d4f537d0f907 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 23 Jan 2023 12:37:49 -0500 Subject: [PATCH 086/222] Adding updates after the implementation Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 481 +++++++++++++++++++++++-------------- 1 file changed, 307 insertions(+), 174 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index ba0c4ec61..dbbc93eca 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -77,63 +77,42 @@ Let's see some examples of the proposed behavior A folder on disk (accessible by the lifecycle) is required to execute the feature, this new folder works as a local registry and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -Lifecycle phases accepts arguments pointing to `image reference`, those arguments are: - -| Input | Description -|-------|------------ -| `` | Tag reference to which the app image will be written | -| `` | Image reference to be analyzed (usually the result of the previous build) | -| `` | Run image reference | - - - -Let's suppose a directory exits and it has the following structure: +Let's suppose a directory exists and it has the following structure: ```=shell -oci -├── cnb -│ ├── my-full-stack-run -│ │ ├── blobs -│ │ │ └── sha256 -│ │ │ ├── 1f59...944a // manifest -│ │ │ ├── 6388...af5a // config -│ │ │ ├── 824b...f984e -│ │ │ ├── f5f7...5b38 -│ │ │ └── 870e...f1b09 -│ │ ├── index.json -│ │ └── oci-layout -│ └── my-partial-stack-run -│ ├── blobs -│ │ └── sha256 -│ │ ├── 1f59...944a // manifest -│ │ └── 6388...af5a // config -│ ├── index.json -│ └── oci-layout -├── foo -│ └── my-cache -│ ├── blobs -│ │ └── sha256 -│ │ └── 1f591..a -│ ├── index.json -│ └── oci-layout -├── bar -│ └── my-previous-app -│ ├── blobs -│ │ └── sha256 -│ │ └── 1efg..w -│ ├── index.json -│ └── oci-layout -└── my-app-image - └── blobs - ├── sha256 - │ ├── 1bcd5..x // app image manifest - │ ├── 2f789..d // app image config - │ ├── 824b...f984e // run layer - │ ├── f5f7...5b38 // run layer - │ ├── 870e...f1b09 // run layer - │ └── 3g234..f // buildpack layer - ├── index.json - └── oci-layout +layout-repo +└── index.docker.io + ├── cnb + │ ├── my-full-stack-run:bionic + │ │ └── bionic + │ │ └── blobs + │ │ ├── sha256 + │ │ │ ├── 1f59...944a // manifest + │ │ │ ├── 6388...af5a // config + │ │ │ ├── 824b...f984e + │ │ │ ├── f5f7...5b38 + │ │ │ └── 870e...f1b09 + │ │ ├── index.json + │ │ └── oci-layout + │ └── my-partial-stack-run:bionic + │ └── bionic + │ ├── blobs + │ │ └── sha256 + │ │ ├── 1f59...944a // manifest + │ │ └── 6388...af5a // config + │ ├── index.json + │ └── oci-layout + └── bar + └── my-previous-app + └── latest + ├── blobs + │ └── sha256 + │ ├── 4bcd5..x // app image manifest + │ ├── 5f789..d // app image config + │ ├── 624b...f984e // run layer + │ └── 4g234..f // buildpack layer + ├── index.json + └── oci-layout ``` The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** represents the same image but the partial one has missing `blobs`, those `blobs` are the layers that are already available in the store from it came from. @@ -150,15 +129,20 @@ In any case the expected output is the same. ##### Analyzing run-image full saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR > /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image +``` -# expected output -# analyzed.toml file with the usual `run-image.reference` +expected analyzed.toml output + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" ``` @@ -171,10 +155,14 @@ In any case the expected output is the same. # OR > /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +``` -# expected output -# analyzed.toml file with the usual `run-image.reference` -# there should be no change behavior in the phase even thought the image has missing blobs +expected analyzed.toml output + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" ``` @@ -187,9 +175,19 @@ In any case the expected output is the same. # OR > /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image +``` + +expected analyzed.toml output + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + +[previous-image] + reference = "" + name = "/layout-repo/index.docker.io/bar/my-previous-app/latest" -# expected output -# analyzed.toml file with the usual `run-image.reference` and `previos-image.reference` ``` ##### Analyzing run-image not saved on disk @@ -204,7 +202,7 @@ In any case the expected output is the same. # expected output -ERROR: the run-image could not be found at path: /oci/cnb/bad-run-image +ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cnb/bad-run-image/latest ``` ##### Analyzing without run-image argument @@ -219,48 +217,54 @@ ERROR: the run-image could not be found at path: /oci/cnb/bad-run-image # expected output -ERROR: -run-image is required when OCI feature is enabled +ERROR: -run-image is required when OCI Layout feature is enabled ``` +Let's also check some examples when the export phase is executed + #### Export phase ##### Export to OCI using run-image full saved on disk ```=shell > export CNB_USE_OCI_LAYOUT=true -> /cnb/lifecycle/exporter -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter my-app-image # OR -> /cnb/lifecycle/exporter -layout -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout my-app-image +``` -# expected output -# the store folder will be updated with the application image as follows - -oci -├── cnb -│ └── my-full-stack-run -│ ├── blobs -│ │ └── sha256 -│ │ ├── 1f59...944a // manifest -│ │ ├── 6388...af5a // config -│ │ ├── 824b...f984e -│ │ ├── f5f7...5b38 -│ │ └── 870e...f1b09 -│ ├── index.json -│ └── oci-layout -├── // some folders omitted for simplicity -└── my-app-image - └── blobs - ├── sha256 - │ ├── 1bcd5..x // app image manifest - │ ├── 2f789..d // app image config - │ ├── 824b...f984e // run layer - │ ├── f5f7...5b38 // run layer - │ ├── 870e...f1b09 // run layer - │ └── 3g234..f // buildpack layer - ├── index.json - └── oci-layout +The output will be written into the repository folder described above and it should looks like this: + +```=shell +layout-repo +└── index.docker.io + ├── cnb + │ └── my-full-stack-run:bionic + │ └── bionic + │ └── blobs + │ ├── sha256 + │ │ ├── 1f59...944a // manifest + │ │ ├── 6388...af5a // config + │ │ ├── 824b...f984e + │ │ ├── f5f7...5b38 + │ │ └── 870e...f1b09 + │ ├── index.json + │ └── oci-layout + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout ``` @@ -271,33 +275,37 @@ As we can see, the application image `my-app-image` contains a **full** copy of ```=shell > export CNB_USE_OCI_LAYOUT=true -> /cnb/lifecycle/exporter -run-image cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter my-app-image # OR -> /cnb/lifecycle/exporter -layout -run-image cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout my-app-image +``` -# expected output -# the store folder will be updated with the application image as follows - -oci -├── cnb -│ └── my-partial-stack-run -│ ├── blobs -│ │ └── sha256 -│ │ ├── 1f59...944a // manifest -│ │ ├── 6388...af5a // config -│ ├── index.json -│ └── oci-layout -├── // some folders omitted for simplicity -└── my-app-image - └── blobs - ├── sha256 - │ ├── 1bcd5..x // app image manifest - │ ├── 2f789..d // app image config - │ └── 3g234..f // buildpack layer - ├── index.json - └── oci-layout +Expected output: + +```=shell +layout-repo +└── index.docker.io + ├── cnb + │ └── my-partial-stack-run:bionic + │ └── bionic + │ ├── blobs + │ │ └── sha256 + │ │ ├── 1f59...944a // manifest + │ │ └── 6388...af5a // config + │ ├── index.json + │ └── oci-layout + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout ``` @@ -338,127 +346,230 @@ The following new inputs are proposed to be added to these phases | Input | Environment Variable | Default Value | Description |-------|-----------------------|---------------|-------------- | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | - | `` | `CNB_OCI_LAYOUT_PATH` | /oci | Path to oci directory where the images are saved | + | `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| + +## How to map an image reference into a path in the layout repository +[rules]: #rules + +In the previous examples one key element was how to translate an image reference into a path to look for in the ``, let's define those rules. + +Considering an **image reference** refers to either a tag reference or digest reference. It could have the following formats +- A tag reference refers to an identifier of form `/:` +- A digest reference refers to a content addressable identifier of form `/@:` -- WHEN `the new flag -layout or the default environment variable CNB_USE_OCI_LAYOUT are set to true` during the phase invocation THEN the feature will be enabled. The image look up will be done following these rules: - WHEN `the image points to a tag reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - WHEN `the image points to a digest reference` - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` - - Apart for the image look up, the logic for each phase should remain the same and the lifecycle will write the layers on disk according to the following rules: - - Any layer generated by a buildpack SHOULD always be written to disk - - Layers from `base-image` or `run-image` MAY be written on disk if they were provided + - WHEN `` is not provided default value will be **index.docker.io** + - IF `` is not also provided, then default value will be **library** -Let's review our previous examples. ## Examples -In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting the new environment variable `CNB_USE_OCI_LAYOUT` to true. +In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting +the new environment variable `CNB_USE_OCI_LAYOUT` to true. + +Let's review some of the previous examples #### Analyze phase ##### Analyzing run-image full saved on disk +Command: + +```=shell +> export CNB_USE_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image +``` + Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/layout-repo` Lifecycle applies the rules for looking up the images: - - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. - - Lifecycle validates the `annotations` map in the **Image Manifest** contains a key-value paired `"org.opencontainers.image.ref.name" : "bionic"` - - In case of the *application image* it will look at path `/oci/my-app-image` + - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and look at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + + - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` - Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. A new field `Name` was added into the `analyzed.toml` that will contain the path resolved by the lifecycle, in these cases: + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + +``` ##### Analyzing run-image partial saved on disk +Command received: + +```=shell +> export CNB_USE_OCI_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +``` + Arguments received: - `run-image`: `cnb/my-full-partial-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/layout-repo` Noticed the structure of the `run-image` provided ```=shell -oci -├── cnb - └── my-partial-stack-run - ├── blobs - │ └── sha256 - │ ├── 1f59...944a // manifest - │ └── 6388...af5a // config - ├── index.json - └── oci-layout +layout-repo +└── index.docker.io + └── cnb + └── my-partial-stack-run:bionic + └── bionic + ├── blobs + │ └── sha256 + │ ├── 1f59...944a // manifest + │ └── 6388...af5a // config + ├── index.json + └── oci-layout ``` -Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/oci/cnb/my-partial-stack-run` and it determines a partial image was provided and execute the phase logic with the information from the **Image Manifest** and the **Image Config** +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase logic with the information from the **Image Manifest** and the **Image Config** + +The output `analyzed.toml` will also include the new `name` field with the path where the image was located. + +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + +``` ##### Analyzing previous-image +Command received: + +```=shell +> export CNB_USE_OCI_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image +``` + Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `previous-image`: `bar/my-previous-app` - `image`: `my-app-image` -The `` is set with the default value `/oci` +The `` is set with the default value `/layout-repo` -`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/oci/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/layout-repo/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -##### Analyzing run-image not saved on disk +The `analyzed.toml` file es expected to be updated with the previous image section and the new label `name` will be also be there with the path to the `previous-image` -Arguments received: +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" -- `run-image`: `cnb/bad-run-image` -- `image`: `my-app-image` +[previous-image] + reference = "" + name = "/layout-repo/index.docker.io/bar/my-previous-app/latest" -The `` is set with the default value `/oci` +``` -In this case, Lifecycle will will look at path `/oci/bad-run-image` and because the path doesn't exists then an error must be thrown. +Let's check how the `export` examples works on detailed -##### Analyzing without run-image argument +##### Export to OCI using run-image full saved on disk -Arguments received: +Pre-conditions: -- `image`: `my-app-image` +The following directories are accessible by the lifecycle +```=shell +/ +├── layout-repo +│ └── index.docker.io +│ └── cnb +│ └── my-full-stack-run:bionic +│ └── bionic +│ └── blobs +│ ├── sha256 +│ │ ├── 1f59...944a // manifest +│ │ ├── 6388...af5a // config +│ │ ├── 824b...f984e +│ │ ├── f5f7...5b38 +│ │ └── 870e...f1b09 +│ ├── index.json +│ └── oci-layout +└── layers + └── analyzed.tom +``` -The `` is set with the default value `/oci` +The `/layers/analyzed.toml` file contains the following data: -When the feature is enabled, Lifecycle requires any input image must be available on disk, because the `run-image` reference was not provided Lifecycle must thrown an error. +```=toml +[run-image] + reference = "" + name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" -#### Export phase +``` -##### Export to OCI using run-image full saved on disk +Command executed: + +```=shell +> export CNB_USE_OCI_LAYOUT=true +> /cnb/lifecycle/exporter my-app-image + +# OR + +> /cnb/lifecycle/exporter -layout my-app-image +``` Arguments received: -- `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/oci` - -Lifecycle applies the rules for looking up the images: - - It takes the **tag reference** `cnb/my-full-stack-run:bionic` and look at path `/oci/cnb/my-full-stack-run` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. - - Lifecycle validates the `annotations` map in the **Image Manifest** contains a key-value paired `"org.opencontainers.image.ref.name" : "bionic"` - - Lifecycle determines the `run-image` is a full image reference (it contains all the blobs) - - Lifecycle writes the *application image* at path `/oci/my-app-image` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format including the layers from the `run-image` +The `` is set with the default value `/layout-repo` -##### Export to OCI using run-image partially saved on disk +Lifecycle: + - It will read the `[run-image]` section in the `analyzed.toml` and read the `name` attribute to load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format -Arguments received: -- `run-image`: `cnb/my-partial-stack-run:bionic` -- `image`: `my-app-image` +The output image will be written at: -The `` is set with the default value `/oci` +```=shell +layout-repo +└── index.docker.io + └── library + └── my-app-image + └── latest + ├── blobs + │ └── sha256 + │ ├── 1bcd5..x // app image manifest + │ ├── 2f789..d // app image config + │ ├── 824b...f984e // run layer + │ ├── f5f7...5b38 // run layer + │ ├── 870e...f1b09 // run layer + │ └── 3g234..f // buildpack layer + ├── index.json + └── oci-layout -Lifecycle behaves similar to the previous example, but during the process of writing the output application image to disk it will skip the layers that are missing in the `blobs` folder at path `/oci/cnb/my-partial-stack-run` +``` ## Proof of concept @@ -570,7 +681,29 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes - +The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#inputs-5) must be updated to include the following inputs to the [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/), [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) and [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) phases + +| Input | Environment Variable | Default Value | Description +|-------|-----------------------|---------------|-------------- +| `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | +| `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| + +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the new `name` field + +```=toml +[image] + reference = "" + name = "" + +[run-image] + reference = "" + name = "" + +[previous-image] + reference = "" + name = "" +``` + +Where + +* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#rules) From f498b0ac99ced52b7ca04335f41f22571d82b2de Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 23 Jan 2023 12:41:24 -0500 Subject: [PATCH 087/222] Adding updates after the implementation Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index dbbc93eca..169a414f3 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -349,7 +349,6 @@ The following new inputs are proposed to be added to these phases | `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| ## How to map an image reference into a path in the layout repository -[rules]: #rules In the previous examples one key element was how to translate an image reference into a path to look for in the ``, let's define those rules. @@ -706,4 +705,4 @@ Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform Where -* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#rules) +* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) From 5d9b008f3f6db79993cbcf0846f52aee8bb2236f Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 23 Jan 2023 12:56:53 -0500 Subject: [PATCH 088/222] Open questions Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 169a414f3..97690fd39 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -672,9 +672,15 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is # Unresolved Questions [unresolved-questions]: #unresolved-questions - # Spec. Changes (OPTIONAL) From 6fc05ff497934e1e2baa9a56900e2030394fa8d3 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 23 Jan 2023 15:22:25 -0500 Subject: [PATCH 089/222] Unresolved question about exporting to tarball Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 97690fd39..fd2b1e3fb 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -676,6 +676,7 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - Tools like [umoci](https://umo.ci/) used to create a runtime bundle from an image in OCI layout format, requires the [annotation](https://github.com/opencontainers/image-spec/blob/main/annotations.md#pre-defined-annotation-keys) `org.opencontainers.image.ref.name` to be present. Also tools like [skopeo](https://github.com/containers/skopeo) when an image is `copy` in oci format the annotation is included. We are not adding the annotation as part of the Buildpacks Specification, but in this case this could make our output incompatible with some other tooling. - Depending on the previous question, the rules for mapping an image reference into a path could change, also validations that the Lifecycle could do to verify the `org.opencontainers.image.ref.name` against the value provided in the image reference. + - Exporting to a tarball can be handle on this RFC or a new one must be created? - What parts of the design do you expect to be resolved through implementation of the feature? - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space From 40cd649d67b3611e18160744821f5baa665a54aa Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 24 Jan 2023 09:39:52 -0500 Subject: [PATCH 090/222] updating the rules to map image ref to path Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index fd2b1e3fb..82724153f 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -353,14 +353,14 @@ The following new inputs are proposed to be added to these phases In the previous examples one key element was how to translate an image reference into a path to look for in the ``, let's define those rules. Considering an **image reference** refers to either a tag reference or digest reference. It could have the following formats -- A tag reference refers to an identifier of form `/:` -- A digest reference refers to a content addressable identifier of form `/@:` +- A tag reference refers to an identifier of form `//:` +- A digest reference refers to a content addressable identifier of form `//@:` The image look up will be done following these rules: - WHEN `the image points to a tag reference` - - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` - WHEN `the image points to a digest reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `///` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` - WHEN `` is not provided default value will be **index.docker.io** - IF `` is not also provided, then default value will be **library** From 1771f9ff51157c185d6eaa6eb5eb341d174ce31d Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 30 Jan 2023 11:55:33 -0500 Subject: [PATCH 091/222] Removing the Name field from the analyzed.toml Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 48 ++++++++++++++++---------------------- 1 file changed, 20 insertions(+), 28 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 82724153f..cfbc869fb 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -141,8 +141,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -161,8 +160,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -181,12 +179,10 @@ expected analyzed.toml output ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "" - name = "/layout-repo/index.docker.io/bar/my-previous-app/latest" + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -399,12 +395,11 @@ Lifecycle applies the rules for looking up the images: - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` -Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. A new field `Name` was added into the `analyzed.toml` that will contain the path resolved by the lifecycle, in these cases: +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -444,14 +439,13 @@ layout-repo └── oci-layout ``` -Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase logic with the information from the **Image Manifest** and the **Image Config** +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** -The output `analyzed.toml` will also include the new `name` field with the path where the image was located. +The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -478,16 +472,14 @@ The `` is set with the default value `/layout-repo` `run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/layout-repo/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -The `analyzed.toml` file es expected to be updated with the previous image section and the new label `name` will be also be there with the path to the `previous-image` +The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "" - name = "/layout-repo/index.docker.io/bar/my-previous-app/latest" + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -522,8 +514,7 @@ The `/layers/analyzed.toml` file contains the following data: ```=toml [run-image] - reference = "" - name = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -545,7 +536,8 @@ Arguments received: The `` is set with the default value `/layout-repo` Lifecycle: - - It will read the `[run-image]` section in the `analyzed.toml` and read the `name` attribute to load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format @@ -694,22 +686,22 @@ The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/p | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | | `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| -Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the new `name` field +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. ```=toml [image] reference = "" - name = "" [run-image] reference = "" - name = "" [previous-image] reference = "" - name = "" ``` Where -* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) +- `[image|run-image|previos-image].reference` MUST be either a digest reference to an image in an OCI registry or the ID of an image in a docker daemon. + - In case an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is being used, it will also include the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) + - The format MUST be as follows: `[path]@[digest]` + From 35bd3e9513454b17c5f29bb7fb5564fd17cff0b8 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Feb 2023 12:49:10 -0500 Subject: [PATCH 092/222] removing the layout-repo flag Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 78 +++++++++++++++----------------------- 1 file changed, 31 insertions(+), 47 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index cfbc869fb..06e7247b8 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -75,13 +75,12 @@ Let's see some examples of the proposed behavior ### Requirements -A folder on disk (accessible by the lifecycle) is required to execute the feature, this new folder works as a local registry and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +Lifecycle will converts image references into local paths following define [rules](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -Let's suppose a directory exists and it has the following structure: +Let's suppose a *platform implementor* creates directories with the following structure: ```=shell -layout-repo -└── index.docker.io +index.docker.io ├── cnb │ ├── my-full-stack-run:bionic │ │ └── bionic @@ -141,7 +140,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -160,7 +159,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-partial-stack-run@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -179,10 +178,10 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + reference = "/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -198,7 +197,7 @@ expected analyzed.toml output # expected output -ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cnb/bad-run-image/latest +ERROR: the run-image could not be found at path: /index.docker.io/cnb/bad-run-image/latest ``` ##### Analyzing without run-image argument @@ -234,8 +233,7 @@ Let's also check some examples when the export phase is executed The output will be written into the repository folder described above and it should looks like this: ```=shell -layout-repo -└── index.docker.io +index.docker.io ├── cnb │ └── my-full-stack-run:bionic │ └── bionic @@ -281,8 +279,7 @@ As we can see, the application image `my-app-image` contains a **full** copy of Expected output: ```=shell -layout-repo -└── index.docker.io +index.docker.io ├── cnb │ └── my-partial-stack-run:bionic │ └── bionic @@ -341,22 +338,21 @@ The following new inputs are proposed to be added to these phases | Input | Environment Variable | Default Value | Description |-------|-----------------------|---------------|-------------- - | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | - | `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| + | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | ## How to map an image reference into a path in the layout repository -In the previous examples one key element was how to translate an image reference into a path to look for in the ``, let's define those rules. +In the previous examples one key element was how to translate an image reference into a path, let's define those rules. Considering an **image reference** refers to either a tag reference or digest reference. It could have the following formats -- A tag reference refers to an identifier of form `//:` +- A name reference refers to an identifier of form `//:` - A digest reference refers to a content addressable identifier of form `//@:` The image look up will be done following these rules: - - WHEN `the image points to a tag reference` - - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` + - WHEN `the image points to a name reference` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` - WHEN `the image points to a digest reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` - WHEN `` is not provided default value will be **index.docker.io** - IF `` is not also provided, then default value will be **library** @@ -388,18 +384,16 @@ Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/layout-repo` - Lifecycle applies the rules for looking up the images: - - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and look at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. + - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and look at path `/index.docker.io/cnb/my-full-stack-run/bionic` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. - - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` + - In case of the *application image* it will look at path `/index.docker.io/library/my-app-image/latest` Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -421,13 +415,10 @@ Arguments received: - `run-image`: `cnb/my-full-partial-run:bionic` - `image`: `my-app-image` -The `` is set with the default value `/layout-repo` - Noticed the structure of the `run-image` provided ```=shell -layout-repo -└── index.docker.io +index.docker.io └── cnb └── my-partial-stack-run:bionic └── bionic @@ -439,13 +430,13 @@ layout-repo └── oci-layout ``` -Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -468,18 +459,16 @@ Arguments received: - `previous-image`: `bar/my-previous-app` - `image`: `my-app-image` -The `` is set with the default value `/layout-repo` - -`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/layout-repo/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +`run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + reference = "/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -492,8 +481,7 @@ Pre-conditions: The following directories are accessible by the lifecycle ```=shell / -├── layout-repo -│ └── index.docker.io +└── index.docker.io │ └── cnb │ └── my-full-stack-run:bionic │ └── bionic @@ -514,7 +502,7 @@ The `/layers/analyzed.toml` file contains the following data: ```=toml [run-image] - reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -533,19 +521,16 @@ Arguments received: - `image`: `my-app-image` -The `` is set with the default value `/layout-repo` - Lifecycle: - - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/index.docker.io/cnb/my-full-stack-run/bionic`. - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. - - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format + - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format The output image will be written at: ```=shell -layout-repo -└── index.docker.io +index.docker.io └── library └── my-app-image └── latest @@ -671,7 +656,7 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - Exporting to a tarball can be handle on this RFC or a new one must be created? - What parts of the design do you expect to be resolved through implementation of the feature? - - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space + - Handle symbolic links to the blobs in the repository, this could be more efficient on hard drive space @@ -684,7 +669,6 @@ The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/p | Input | Environment Variable | Default Value | Description |-------|-----------------------|---------------|-------------- | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | -| `` | `CNB_OCI_LAYOUT_PATH` | /layout-repo | Path to a directory where the images are saved in OCI layout format| Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. From eba5b7e9a9cc17fd7012df975edf4989bee951df Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 8 Feb 2023 17:26:35 -0500 Subject: [PATCH 093/222] Revert "Removing the Name field from the analyzed.toml" This reverts commit 9e5c2d0ba198112d0e808b4f04a25937287e48cc. Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 48 +++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 27 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 06e7247b8..53f70e8a6 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -140,7 +140,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -159,7 +159,7 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/index.docker.io/cnb/my-partial-stack-run@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -178,10 +178,10 @@ expected analyzed.toml output ```=toml [run-image] - reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -336,8 +336,8 @@ Notice that we are relying on the OCI format Specification to expose the data fo The following new inputs are proposed to be added to these phases - | Input | Environment Variable | Default Value | Description - |-------|-----------------------|---------------|-------------- + | Input | Environment Variable | Default Value | Description | + |-------|-----------------------|---------------|-------------| | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | ## How to map an image reference into a path in the layout repository @@ -389,12 +389,11 @@ Lifecycle applies the rules for looking up the images: - In case of the *application image* it will look at path `/index.docker.io/library/my-app-image/latest` -Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. A new field `Name` was added into the `analyzed.toml` that will contain the path resolved by the lifecycle, in these cases: ```=toml [run-image] - reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" - + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` ##### Analyzing run-image partial saved on disk @@ -430,14 +429,12 @@ index.docker.io └── oci-layout ``` -Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** - -The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. +Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** +The output `analyzed.toml` will also include the new `name` field with the path where the image was located. ```=toml [run-image] - reference = "/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" - + reference = "/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` ##### Analyzing previous-image @@ -461,14 +458,14 @@ Arguments received: `run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` +The `analyzed.toml` file es expected to be updated with the previous image section and the new label `name` will be also be there with the path to the `previous-image` ```=toml [run-image] - reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" [previous-image] - reference = "/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" + reference = "/layout-repo/index.docker.io/bar/my-previous-app/latest@sha256:aa0cf7fc8f161bdb96166c1644174affacd70d17f372373ca72c8e91116e2d43" ``` @@ -502,7 +499,7 @@ The `/layers/analyzed.toml` file contains the following data: ```=toml [run-image] - reference = "/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" + reference = "/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic@sha256:fab3bb83de466ed29d7e9dcfdbee5b5fb2ff90e91bc849af85b261b4c2062a7a" ``` @@ -522,9 +519,9 @@ Arguments received: - `image`: `my-app-image` Lifecycle: - - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/index.docker.io/cnb/my-full-stack-run/bionic`. + - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. - - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format + - Lifecycle will execute the export steps and at the end of the process it will write the *application image* at path `/layout-repo/index.docker.io/library/my-app-image/latest` in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format The output image will be written at: @@ -666,11 +663,11 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#inputs-5) must be updated to include the following inputs to the [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/), [Analyze](https://buildpacks.io/docs/concepts/components/lifecycle/analyze/) and [Export](https://buildpacks.io/docs/concepts/components/lifecycle/export/) phases -| Input | Environment Variable | Default Value | Description -|-------|-----------------------|---------------|-------------- +| Input | Environment Variable | Default Value | Description| +|-------|-----------------------|---------------|------------| | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | -Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the new `name` field ```=toml [image] @@ -685,7 +682,4 @@ Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform Where -- `[image|run-image|previos-image].reference` MUST be either a digest reference to an image in an OCI registry or the ID of an image in a docker daemon. - - In case an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is being used, it will also include the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) - - The format MUST be as follows: `[path]@[digest]` - +* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) From 1086f453a927260743f7d4a70a0e9e3c591d4e2e Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 8 Feb 2023 18:27:57 -0500 Subject: [PATCH 094/222] adding Natalie's feedback Signed-off-by: Juan Bustamante Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 131 ++++++++++++++++++++++++------------- 1 file changed, 87 insertions(+), 44 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 53f70e8a6..3cb550a98 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -77,10 +77,11 @@ Let's see some examples of the proposed behavior Lifecycle will converts image references into local paths following define [rules](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) and the content must be in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -Let's suppose a *platform implementor* creates directories with the following structure: +Let's suppose a *platform implementor* creates a directory with the following structure: ```=shell -index.docker.io +layout-repo +└── index.docker.io ├── cnb │ ├── my-full-stack-run:bionic │ │ └── bionic @@ -118,8 +119,8 @@ The images named **cnb/my-full-stack-run** and **cnb/my-partial-stack-run** repr For each example case, I will present two ways of enabling the new capability: -- Using an environment Variable -- Using a new `-layout` flag +- Using an environment variables +- Using the new `-layout` and `layout-dir` flags In any case the expected output is the same. @@ -129,11 +130,12 @@ In any case the expected output is the same. ```=shell > export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic my-app-image ``` expected analyzed.toml output @@ -147,12 +149,13 @@ expected analyzed.toml output ##### Analyzing run-image partial saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image ``` expected analyzed.toml output @@ -166,12 +169,13 @@ expected analyzed.toml output ##### Analyzing previous-image ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic-previous-image bar/my-previous-app my-app-image ``` expected analyzed.toml output @@ -188,33 +192,48 @@ expected analyzed.toml output ##### Analyzing run-image not saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/bad-run-image my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/bad-run-image my-app-image # expected output -ERROR: the run-image could not be found at path: /index.docker.io/cnb/bad-run-image/latest +ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cnb/bad-run-image/latest ``` ##### Analyzing without run-image argument ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer my-app-image # OR -> /cnb/lifecycle/analyzer -layout my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo my-app-image # expected output ERROR: -run-image is required when OCI Layout feature is enabled ``` +```=shell +> export CNB_USE_LAYOUT=true +> /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image + +# OR + +> /cnb/lifecycle/analyzer -layout -run-image cnb/bad-run-image my-app-image + +# expected output + +ERROR: defining a layout directory is required when OCI Layout feature is enabled. Use -layout-dir flag or CNB_LAYOUT_DIR environment variable +``` + Let's also check some examples when the export phase is executed #### Export phase @@ -222,18 +241,20 @@ Let's also check some examples when the export phase is executed ##### Export to OCI using run-image full saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/exporter my-app-image # OR -> /cnb/lifecycle/exporter -layout my-app-image +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo my-app-image ``` The output will be written into the repository folder described above and it should looks like this: ```=shell -index.docker.io +layout-repo +└── index.docker.io ├── cnb │ └── my-full-stack-run:bionic │ └── bionic @@ -268,18 +289,20 @@ As we can see, the application image `my-app-image` contains a **full** copy of ##### Export to OCI using run-image partially saved on disk ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/exporter my-app-image # OR -> /cnb/lifecycle/exporter -layout my-app-image +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo my-app-image ``` Expected output: ```=shell -index.docker.io +layout-repo +└── index.docker.io ├── cnb │ └── my-partial-stack-run:bionic │ └── bionic @@ -309,12 +332,13 @@ As we can see, the application image `my-app-image` has missing `blobs` because Any combination of using multiple sources or sinks in the Lifecycle invocation of phases should throw an error to the user. For example: ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/exporter -daemon -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/exporter -layout -daemon -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/exporter -layout -layout-dir /layout-repo -daemon -run-image cnb/my-full-stack-run:bionic my-app-image ERROR: exporting to multiples target is not allowed ``` @@ -336,9 +360,10 @@ Notice that we are relying on the OCI format Specification to expose the data fo The following new inputs are proposed to be added to these phases - | Input | Environment Variable | Default Value | Description | - |-------|-----------------------|---------------|-------------| - | `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | + | Input | Environment Variable | Default Value | Description + |-------|-----------------------|---------------|-------------- + | `` | `CNB_USE_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | + | `` | `CNB_LAYOUT_DIR` | | Path to a directory where the images are saved in OCI layout format| ## How to map an image reference into a path in the layout repository @@ -360,9 +385,9 @@ The image look up will be done following these rules: ## Examples In all the examples the new feature is enabled by the use of the new flag `-layout` or by setting -the new environment variable `CNB_USE_OCI_LAYOUT` to true. +the new environment variable `CNB_USE_LAYOUT` to true. -Let's review some of the previous examples +Let's review some previous examples #### Analyze phase @@ -372,11 +397,12 @@ Command: ```=shell > export CNB_USE_LAYOUT=true +> export CNB_LAYOUT_DIR=/layout-repo > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic my-app-image # OR -> /cnb/lifecycle/analyzer -layout -run-image cnb/my-full-stack-run:bionic my-app-image +> /cnb/lifecycle/analyzer -layout -layout-dir /layout-repo -run-image cnb/my-full-stack-run:bionic my-app-image ``` Arguments received: @@ -384,12 +410,15 @@ Arguments received: - `run-image`: `cnb/my-full-stack-run:bionic` - `image`: `my-app-image` -Lifecycle applies the rules for looking up the images: - - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and look at path `/index.docker.io/cnb/my-full-stack-run/bionic` for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. +The `` is set with the value `/layout-repo` - - In case of the *application image* it will look at path `/index.docker.io/library/my-app-image/latest` +Lifecycle applies the rules for looking up the images: + - It takes the **tag reference** `cnb/my-full-stack-run:bionic`, applies the conversion rules and gets `/index.docker.io/cnb/my-full-stack-run/bionic` + - It will append the `` at the beginning, getting the following path: `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic` + - It will look for an image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. + - In case of the *application image* it will look at path `/layout-repo/index.docker.io/library/my-app-image/latest` -Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. A new field `Name` was added into the `analyzed.toml` that will contain the path resolved by the lifecycle, in these cases: +Because both images are found, the phase is executed as usual and the `analyzed.toml` file will be updated. The `run-image.reference` added into the `analyzed.toml` will contain the path resolved by the lifecycle plus the digest reference to the image with the following format `[path]@[digest]`. In case of this example, it will look like this: ```=toml [run-image] @@ -401,7 +430,7 @@ Because both images are found, the phase is executed as usual and the `analyzed. Command received: ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/cnb/my-partial-stack-run:bionic my-app-image # OR @@ -414,10 +443,13 @@ Arguments received: - `run-image`: `cnb/my-full-partial-run:bionic` - `image`: `my-app-image` +The `` is set with the default value `/layout-repo` + Noticed the structure of the `run-image` provided ```=shell -index.docker.io +layout-repo +└── index.docker.io └── cnb └── my-partial-stack-run:bionic └── bionic @@ -430,7 +462,8 @@ index.docker.io ``` Similar to the previous example, Lifecycle applies the rules for looking up the images and look at path `/layout-repo/index.docker.io/cnb/my-partial-stack-run/bionic` and it determines a partial image was provided and execute the phase with the information from the **Image Manifest** and the **Image Config** -The output `analyzed.toml` will also include the new `name` field with the path where the image was located. + +The output `analyzed.toml` will also include the new `run-image.reference` field the path and the digest of the run image. ```=toml [run-image] @@ -442,7 +475,7 @@ The output `analyzed.toml` will also include the new `name` field with the path Command received: ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/my-full-stack-run:bionic -previous-image bar/my-previous-app my-app-image # OR @@ -456,9 +489,11 @@ Arguments received: - `previous-image`: `bar/my-previous-app` - `image`: `my-app-image` +The `` is set with the default value `/layout-repo` + `run-image` and `image` arguments are treated in the same way as previous examples, and for `previous-image` argument the looking up images rules are applied and Lifecycle will look at path `/index.docker.io/bar/my-previous-app` for a image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format. -The `analyzed.toml` file es expected to be updated with the previous image section and the new label `name` will be also be there with the path to the `previous-image` +The `analyzed.toml` file es expected to be updated with the `previous-image.reference` containing the path and the digest of the `previous-image` ```=toml [run-image] @@ -478,7 +513,8 @@ Pre-conditions: The following directories are accessible by the lifecycle ```=shell / -└── index.docker.io +├── layout-repo +│ └── index.docker.io │ └── cnb │ └── my-full-stack-run:bionic │ └── bionic @@ -506,7 +542,7 @@ The `/layers/analyzed.toml` file contains the following data: Command executed: ```=shell -> export CNB_USE_OCI_LAYOUT=true +> export CNB_USE_LAYOUT=true > /cnb/lifecycle/exporter my-app-image # OR @@ -518,6 +554,8 @@ Arguments received: - `image`: `my-app-image` +The `` is set with the default value `/layout-repo` + Lifecycle: - It will read the `[run-image]` section in the `analyzed.toml`, it will parse `reference` attribute using the `@` separator and load the `run-image` image saved on disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at path `/layout-repo/index.docker.io/cnb/my-full-stack-run/bionic`. - Lifecycle could also validate the digest of the image loaded is the same as the one established by the `reference`. @@ -527,7 +565,8 @@ Lifecycle: The output image will be written at: ```=shell -index.docker.io +layout-repo +└── index.docker.io └── library └── my-app-image └── latest @@ -653,7 +692,7 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - Exporting to a tarball can be handle on this RFC or a new one must be created? - What parts of the design do you expect to be resolved through implementation of the feature? - - Handle symbolic links to the blobs in the repository, this could be more efficient on hard drive space + - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space @@ -665,9 +704,10 @@ The [Platform Interface Specification](https://github.com/buildpacks/spec/blob/p | Input | Environment Variable | Default Value | Description| |-------|-----------------------|---------------|------------| -| `` | `CNB_USE_OCI_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | +| `` | `CNB_USE_LAYOUT` | false | Enables the capability of resolving image from/to in OCI layout format on disk | +| `` | `CNB_LAYOUT_DIR` | | Path to a directory where the images are saved in OCI layout format| -Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the new `name` field +Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform/0.11/platform.md#analyzedtoml-toml) will be updated to include the `reference` format in case of layout is being used. ```=toml [image] @@ -682,4 +722,7 @@ Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform Where -* `[image|run-image|previos-image].name` MUST point to the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) +- `[image|run-image|previos-image].reference` MUST be either a digest reference to an image in an OCI registry or the ID of an image in a docker daemon. + - In case an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is being used, it will also include the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) + - The format MUST be as follows: `[path]@[digest]` + From c1f4f63ef7f3a4348de6a63c2c7bb7533df679c3 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Tue, 14 Feb 2023 18:28:46 -0600 Subject: [PATCH 095/222] RFC 0118 [#272] Signed-off-by: Terence Lee --- text/{0000-2023H1-roadmap.md => 0118-2023H1-roadmap.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-2023H1-roadmap.md => 0118-2023H1-roadmap.md} (98%) diff --git a/text/0000-2023H1-roadmap.md b/text/0118-2023H1-roadmap.md similarity index 98% rename from text/0000-2023H1-roadmap.md rename to text/0118-2023H1-roadmap.md index 2c5538926..38b2241d3 100644 --- a/text/0000-2023H1-roadmap.md +++ b/text/0118-2023H1-roadmap.md @@ -3,10 +3,10 @@ - Name: 2023H1 Roadmap - Start Date: 2023-01-24 - Author(s): hone -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [#272](https://github.com///pull/272) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Summary From 326e7b16bb0dd1bec07c215ed172d6f6381eb550 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Tue, 14 Feb 2023 18:33:14 -0600 Subject: [PATCH 096/222] fix RFC link Signed-off-by: Terence Lee --- text/0118-2023H1-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0118-2023H1-roadmap.md b/text/0118-2023H1-roadmap.md index 38b2241d3..ca582b201 100644 --- a/text/0118-2023H1-roadmap.md +++ b/text/0118-2023H1-roadmap.md @@ -4,7 +4,7 @@ - Start Date: 2023-01-24 - Author(s): hone - Status: Approved -- RFC Pull Request: [#272](https://github.com///pull/272) +- RFC Pull Request: [#272](https://github.com/buildpacks/rfcs/pull/272) - CNB Pull Request: (leave blank) - CNB Issue: - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) From b09fea05ebd6c039e5c2acb9cdfd03202f3e796d Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 15 Feb 2023 11:58:46 -0500 Subject: [PATCH 097/222] adding layout-dir to some missing places Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 3cb550a98..9bcd5fccd 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -221,6 +221,8 @@ ERROR: the run-image could not be found at path: /layout-repo/index.docker.io/cn ERROR: -run-image is required when OCI Layout feature is enabled ``` +##### Analyzing without layout-dir argument + ```=shell > export CNB_USE_LAYOUT=true > /cnb/lifecycle/analyzer -run-image cnb/bad-run-image my-app-image @@ -375,9 +377,9 @@ Considering an **image reference** refers to either a tag reference or digest re The image look up will be done following these rules: - WHEN `the image points to a name reference` - - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` + - Lifecycle will load/save the image from/to disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `////` - WHEN `the image points to a digest reference` - - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` + - Lifecycle will load the image from disk in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format at `/////` - WHEN `` is not provided default value will be **index.docker.io** - IF `` is not also provided, then default value will be **library** From 76d034f4c6ff2470e85b38d2310d343f97d950f2 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 15 Feb 2023 16:37:34 -0500 Subject: [PATCH 098/222] updating the unresolved questions Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index 9bcd5fccd..c96850128 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -690,12 +690,13 @@ I think, this PoC demonstrate that adding the exporting to OCI layout format is - What parts of the design do you expect to be resolved before this gets merged? - Tools like [umoci](https://umo.ci/) used to create a runtime bundle from an image in OCI layout format, requires the [annotation](https://github.com/opencontainers/image-spec/blob/main/annotations.md#pre-defined-annotation-keys) `org.opencontainers.image.ref.name` to be present. Also tools like [skopeo](https://github.com/containers/skopeo) when an image is `copy` in oci format the annotation is included. We are not adding the annotation as part of the Buildpacks Specification, but in this case this could make our output incompatible with some other tooling. - - Depending on the previous question, the rules for mapping an image reference into a path could change, also validations that the Lifecycle could do to verify the `org.opencontainers.image.ref.name` against the value provided in the image reference. - - Exporting to a tarball can be handle on this RFC or a new one must be created? + - **Answer:** we agreed on adding `org.opencontainers.image.ref.name` annotation + - Exporting to a tarball can be handled on this RFC or a new one must be created? + - **Answer:** this can be handled on a different RFC - What parts of the design do you expect to be resolved through implementation of the feature? - Handle symbolic links to the blobs in the `` repository, this could be more efficient on hard drive space - + - **Answer:** this can be handled on the implementation side From 77c9fcce82e1504d29d119e6444e8d5d001e3ddd Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 15 Feb 2023 16:44:18 -0500 Subject: [PATCH 099/222] updating the spec changes Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index c96850128..dc4e70dbd 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -725,7 +725,8 @@ Also the `analyzed.toml` [file](https://github.com/buildpacks/spec/blob/platform Where -- `[image|run-image|previos-image].reference` MUST be either a digest reference to an image in an OCI registry or the ID of an image in a docker daemon. - - In case an image in [OCI Image Layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) format is being used, it will also include the path of the image in OCI layout format following the rules describe [previously](#how-to-map-an-image-reference-into-a-path-in-the-layout-repository) - - The format MUST be as follows: `[path]@[digest]` +- `[image|run-image|previos-image].reference` MUST be either: + - A digest reference to an image in an OCI registry + - The ID of an image in a docker daemon + - The path to an image in OCI layout format From 8beb9e83ba5efc77bb26985dbd2a2daccd6199ec Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 17 Feb 2023 14:47:56 -0500 Subject: [PATCH 100/222] Update 0000-export-to-oci.md Fixing error message according to the implementation Signed-off-by: Juan Bustamante --- text/0000-export-to-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-export-to-oci.md b/text/0000-export-to-oci.md index dc4e70dbd..d1b806f22 100644 --- a/text/0000-export-to-oci.md +++ b/text/0000-export-to-oci.md @@ -342,7 +342,7 @@ Any combination of using multiple sources or sinks in the Lifecycle invocation o > /cnb/lifecycle/exporter -layout -layout-dir /layout-repo -daemon -run-image cnb/my-full-stack-run:bionic my-app-image -ERROR: exporting to multiples target is not allowed +ERROR: exporting to multiple targets is unsupported ``` # How it Works From b533d041d70fcfe53be251b9710d0781edb45d5c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 20 Feb 2023 01:05:56 +0000 Subject: [PATCH 101/222] Bump kentaro-m/auto-assign-action from 1.2.4 to 1.2.5 Bumps [kentaro-m/auto-assign-action](https://github.com/kentaro-m/auto-assign-action) from 1.2.4 to 1.2.5. - [Release notes](https://github.com/kentaro-m/auto-assign-action/releases) - [Commits](https://github.com/kentaro-m/auto-assign-action/compare/v1.2.4...v1.2.5) --- updated-dependencies: - dependency-name: kentaro-m/auto-assign-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- .github/workflows/auto-assign-maintainer.yml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/auto-assign-maintainer.yml b/.github/workflows/auto-assign-maintainer.yml index 80b2ec846..292bae4e7 100644 --- a/.github/workflows/auto-assign-maintainer.yml +++ b/.github/workflows/auto-assign-maintainer.yml @@ -14,7 +14,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.4 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-core.yml distribution: @@ -23,7 +23,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.4 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-distribution.yml implementation: @@ -32,7 +32,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.4 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-implementation.yml learning: @@ -41,7 +41,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.4 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-learning.yml platform: @@ -50,7 +50,7 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.4 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-platform.yml bat: @@ -59,6 +59,6 @@ jobs: runs-on: - ubuntu-latest steps: - - uses: kentaro-m/auto-assign-action@v1.2.4 + - uses: kentaro-m/auto-assign-action@v1.2.5 with: configuration-path: .github/auto-assign-bat.yml From d18479bbffe6e080cca90884e399f68be27fbd47 Mon Sep 17 00:00:00 2001 From: Joe Kimmel <86852107+joe-kimmel-vmw@users.noreply.github.com> Date: Mon, 6 Mar 2023 14:39:52 -0700 Subject: [PATCH 102/222] Update 0096-remove-stacks-mixins.md Signed-off-by: Joe Kimmel <86852107+joe-kimmel-vmw@users.noreply.github.com> --- text/0096-remove-stacks-mixins.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/text/0096-remove-stacks-mixins.md b/text/0096-remove-stacks-mixins.md index 31dee5d68..34f17ea98 100644 --- a/text/0096-remove-stacks-mixins.md +++ b/text/0096-remove-stacks-mixins.md @@ -59,8 +59,8 @@ For Windows-based images, Distribution should be empty. Version should be the [s The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS, Architecture, and Variant. -If the `targets` list is empty and `/bin/build` is present, a target with `os = "linux"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. -If the `targets` list is empty and `/bin/build.bat` or `/bin/build.exe` is present, a target with `os = "windows"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. +If the `targets` list is empty and `/bin/build` is present, a target with `os = "linux"` and `arch = "amd64"` is assumed by tools reading `buildpack.toml`. +If the `targets` list is empty and `/bin/build.bat` or `/bin/build.exe` is present, a target with `os = "windows"` and `arch = "amd64"` is assumed by tools reading `buildpack.toml`. App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. @@ -73,14 +73,14 @@ When an app image is rebased, `rebaser` must fail if the new run image and previ ```toml [[targets]] os = "linux" -arch = "x86_64" +arch = "amd64" [[targets.distributions]] name = "ubuntu" versions = ["18.04", "20.04"] [[targets]] os = "linux" -arch = "x86_64" +arch = "amd64" [[targets.distributions]] name = "ubuntu" versions = ["14.04", "16.04"] @@ -152,7 +152,7 @@ If the newly-specified field values are missing, the lifecycle and pack may used ``` config.os = "linux" -config.architecture = "x86_64" +config.architecture = "amd64" io.buildpacks.distribution.name = "ubuntu" io.buildpacks.distribution.version = "18.04" ``` @@ -169,7 +169,7 @@ to ```toml [[targets]] os = "linux" -arch = "x86_64" +arch = "amd64" [[targets.distributions]] name = "ubuntu" versions = ["18.04"] @@ -201,3 +201,12 @@ versions = ["18.04"] [spec-changes]: #spec-changes This RFC requires extensive changes to all specifications. + +## Amended +### Summary + +rename x86_64 -> amd64 in keeping with all other usages of arch. descriptors. + +### Motivation + +This is how we do it everywhere else, this is the way. From 49f1700110e6c9e66bf87531ded7dd0fb2c49d04 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Tue, 7 Mar 2023 16:01:44 -0500 Subject: [PATCH 103/222] RFC 0119 [#203] Signed-off-by: Natalie Arellano --- text/{0000-export-to-oci.md => 0119-export-to-oci.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-export-to-oci.md => 0119-export-to-oci.md} (99%) diff --git a/text/0000-export-to-oci.md b/text/0119-export-to-oci.md similarity index 99% rename from text/0000-export-to-oci.md rename to text/0119-export-to-oci.md index d1b806f22..2dacbc55e 100644 --- a/text/0000-export-to-oci.md +++ b/text/0119-export-to-oci.md @@ -3,10 +3,10 @@ - Name: Export to OCI format - Start Date: 2022-02-22 - Author(s): Juan Bustamante (@jjbustamante) -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#203](https://github.com/buildpacks/rfcs/pull/203) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: N/A - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Summary From b01e57faa84914ca4285062f43ac52c947aee888 Mon Sep 17 00:00:00 2001 From: Joe Kimmel Date: Wed, 8 Mar 2023 20:28:57 -0700 Subject: [PATCH 104/222] first draft of CVE Backporting rfc Signed-off-by: Joe Kimmel --- text/0000-cvebackports.md | 118 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 text/0000-cvebackports.md diff --git a/text/0000-cvebackports.md b/text/0000-cvebackports.md new file mode 100644 index 000000000..12e0ef921 --- /dev/null +++ b/text/0000-cvebackports.md @@ -0,0 +1,118 @@ +# Meta +[meta]: #meta +- Name: CVE discretional Patching +- Start Date: 2023-03-08 +- Author(s): joe-kimmel-vmw, natalieparellano +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: N/A + +# Summary +[summary]: #summary +This RFC describes how maintainers MAY issue patch releases in response to critical and high severity CVEs being detected in past or current releases of the lifecycle binary. + +# Definitions +[definitions]: #definitions +CVE - Literally expands to “Common Vulnerabilities and Exposures” but in general it refers to a security gap which could be exploited by a malicious attacker, which can be fixed by patching a single component. + +CVE Severity: CVEs are announced with a severity score. While this score can vary between vendors, it is made by considering many factors including how easy it is to take advantage of the exploit, what resources are exposed by the exploit, and whether there is already a known exploit in circulation. See https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator + +Critical and High are the two highest severity ratings available under the current severity rating system. + + +# Motivation +[motivation]: #motivation + +- Why should we do this? + +Patching CVEs is industry best practice. By providing patched updates of recent releases we enable CNB-using organizations with low risk tolerance or high upgrade friction to make a minimal change to their runtime infrastructure that still keeps secure against known vulnerabilities. +Following patch version line architecture also boosts user trust vs. a pledge to roll forward security fixes to higher version lines with the promise of no breaking changes, which would be looked upon skeptically by some enterprise users. Those same users who are skeptical of any & all promises that new versions won’t have any regressions are especially likely to be running older versions in the wild, thus providing patches to older versions ensures that they can continue to use CNB lifecycle without risking exposure to the riskiest CVEs. + +- What use cases does it support? + +This is especially important for users who do not use CNB as part of a SaaS product, i.e. for “on-prem” deployments. These on-prem users are pulling down the provided lifecycle images and running the binaries on their own infrastructure, thus increasing their potential risk exposure and liability. + +- What is the expected outcome? + +Maintainers MAY issue patch releases in response to critical and high CVEs. Most importantly, users MAY upgrade to consume these patch releases in a timely manner if they are not comfortable consuming the latest version. + +# What it is +[what-it-is]: #what-it-is + +It is risk mitigation for consumer and enterprise customers who want to apply critical patches without any other changes that would come with a minor version upgrade. + +# How it Works +[how-it-works]: #how-it-works + +Patch releases will be published at the discretion of the maintainers in response to Critical and High CVEs. + +The lifecycle will still offer the same strong backwards compatibility guarantees as ever. + +Existing process (patch most recent version N until it becomes N-1) bumps dependencies and the go version each month irrespective of CVEs being present. This proposal does not involve changing that process. + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + +Maintaining past releases does take time. However under this proposal that maintenance is optional and performed at the discretion of the maintainers. + +For a vague guess at the volume of this work: In the year from March 2022 March 2023, the grype scanner found 3 High and 0 Critical CVEs in the 0.13.5 lifecycle binary, so there would have been at most 3 additional patch releases of that line. +Similarly, grype scanner found 1 High and 0 Critical CVEs in the 0.14.3 lifecycle image (from October 2022), and 0 High or Critical CVEs in the 0.15.3 lifecycle image (from Jan 2023). + + + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + - Only Patch the Most Recent Release: This is a fine idea but it only works in a world where all users are willing to migrate to the latest release. This RFC addresses the wants and needs of users who are not comfortable performing minor version upgrades, even with backwards compatibility guarantees, in order to address CVEs. + - Only Patch development trunk, and wait for the next release: This also only works in a world where minor version upgrades are seen as cheap or low-risk to perform. + - Don’t Patch CVEs: we probably wouldn’t ever do this option. + + +- Why is this proposal the best? +This is the only proposal palatable to orgs that view minor releases as high-risk and/or expensive. +- What is the impact of not doing this? +We risk alienating some enterprise users of CNB. + +# Prior Art +[prior-art]: #prior-art + +N/A + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A beyond general feedback and consensus. + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +N/A + +# History +[history]: #history + + From 1019f08613d6b20afd2211438164ce47136f4e9b Mon Sep 17 00:00:00 2001 From: Joe Kimmel Date: Thu, 9 Mar 2023 09:42:20 -0700 Subject: [PATCH 105/222] one more note re. motivation Signed-off-by: Joe Kimmel --- text/0000-cvebackports.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/text/0000-cvebackports.md b/text/0000-cvebackports.md index 12e0ef921..ce02872bf 100644 --- a/text/0000-cvebackports.md +++ b/text/0000-cvebackports.md @@ -34,6 +34,9 @@ Following patch version line architecture also boosts user trust vs. a pledge to This is especially important for users who do not use CNB as part of a SaaS product, i.e. for “on-prem” deployments. These on-prem users are pulling down the provided lifecycle images and running the binaries on their own infrastructure, thus increasing their potential risk exposure and liability. +Additionally, as we do have concrete plans to deprecate platform API versions in the lifecycle this calendar year, there's increased likelihood to learn of +other lifecycle consumers who are not ready to upgrade and who would appreciate patch releases. + - What is the expected outcome? Maintainers MAY issue patch releases in response to critical and high CVEs. Most importantly, users MAY upgrade to consume these patch releases in a timely manner if they are not comfortable consuming the latest version. From d807cc425f08df5d3feb64e566681d9ec3eaf91b Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 11 Apr 2023 12:39:50 -0500 Subject: [PATCH 106/222] Adding buildpack community organization criteria table Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 50 ++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 13 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 1f8d82462..657729844 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -11,7 +11,30 @@ # Summary -This RFC proposes the donation of the open-source project [kpack](https://github.com/pivotal/kpack/) into the [Cloud Native Buildpack](https://buildpacks.io/) (CNB) ecosystem. +This RFC proposes the donation of the open-source project [kpack](https://github.com/pivotal/kpack/) into the [Cloud Native Community Organization Buildpack](https://github.com/buildpacks-community). + +Following the process defined in the [Buildpack Commnity RFC](https://github.com/buildpacks/rfcs/blob/main/text/0117-buildpacks-community.md) the following table presents the criteria used to evaluate the project. + +| Criteria | Evidence | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------| +| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | +| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | +| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | +| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149/files) | +| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | +| The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. || +| The list of project maintainers must be publicly available and controlled through a Github team. || +| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | +| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section | +| The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). || +| There should have visible automated testing for all repositories that are part of the project. || +| The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. || +| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149/files) | +| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. || +| The has a file - ADOPTERS.md: A list of adopters of the project. | See Adopters [Pull Request](https://github.com/pivotal/kpack/pull/1142) | +| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See Versioning [Pull Request](https://github.com/pivotal/kpack/pull/1150) | +| The has a file - RELEASE.md: A guide to how releases are done for the project. || +| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149/files) | # Definitions @@ -307,26 +330,27 @@ Based on the [CNB governance policy](https://github.com/buildpacks/community/blo How do migrate roles and responsibilities into the CNB governance process? -Currently the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team) already has a **team lead** assigned and, by definition, each team can have only one **team lead**. In order to provide the current [kpack](https://github.com/pivotal/kpack/) team with the same accountability for the migrated repositories the proposal is to follow the guidelines describe on the [Component Maintainer Role RFC](https://github.com/buildpacks/rfcs/pull/234) +Currently, the [CNB Platform Team](https://github.com/buildpacks/community/blob/main/TEAMS.md#Platform-Team) already has a **team lead** assigned and, by definition, each team can have only one **team lead**. In order to provide the current [kpack](https://github.com/pivotal/kpack/) team with the same accountability for the migrated repositories the proposal is to follow the guidelines describe on the [Component Maintainer Role RFC](https://github.com/buildpacks/rfcs/pull/234) The [kpack's](https://github.com/pivotal/kpack/) maintainers that will be nominated as **component maintainer** in CNB are: -| Name | Github account | Organization | -| --- | --- | --- | -| Matthew McNew| [@matthewmcnew](https://github.com/matthewmcnew)| VMware| -| Tom Kennedy | [@tomkennedy513](https://github.com/tomkennedy513) | VMware | -| Yael Harel | [@yaelharel](https://github.com/yaelharel) | VMware| -| Daniel Chen |[@chenbh](https://github.com/chenbh) | VMware| -| Juan Bustamante |[@jjbustamante](https://github.com/jjbustamante) | VMware| +| Name | Github account | Organization | +|------------------|----------------------------------------------------|--------------| +| Matthew McNew | [@matthewmcnew](https://github.com/matthewmcnew) | VMware | +| Tom Kennedy | [@tomkennedy513](https://github.com/tomkennedy513) | VMware | +| Nicholas Carlson | [@ncarlson](https://github.com/ncarlson) | VMWare | +| Yael Harel | [@yaelharel](https://github.com/yaelharel) | VMware | +| Daniel Chen | [@chenbh](https://github.com/chenbh) | VMware | +| Juan Bustamante | [@jjbustamante](https://github.com/jjbustamante) | VMware | Also those members are willing to become more involved with CNB projects and become **Platform maintainers** in the near future. Outside VMware, the following contributors manifested their desired to become [kpack's](https://github.com/pivotal/kpack/) **component maintainer**. -| Name | Github account | Organization | -| --- | --- | --- | -| Sambhav Kothari| [@samj1912](https://github.com/samj1912) | Bloomberg | -| Aidan Delaney| [@AidanDelaney](https://github.com/AidanDelaney) | Bloomberg | +| Name | Github account | Organization | +|-----------------|--------------------------------------------------|--------------| +| Sambhav Kothari | [@samj1912](https://github.com/samj1912) | Bloomberg | +| Aidan Delaney | [@AidanDelaney](https://github.com/AidanDelaney) | Bloomberg | #### RFC process From 0a7b1d1a2f9ba32e408f2c640969fc18566f36b9 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 11 Apr 2023 12:42:00 -0500 Subject: [PATCH 107/222] removing adopters from RFC and pointing to the kpack repo Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 657729844..8a20ee3f4 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -264,17 +264,6 @@ pie showData "Others" : 37 ``` -# Adopters - -[VMware](https://www.vmware.com/) is the most obvious company using [kpack](https://github.com/pivotal/kpack/) as a core component in their [Tanzu](https://tanzu.vmware.com/tanzu) offer. [kpack](https://github.com/pivotal/kpack/) has been used in the enterprise for years in hundreds of costumers. - -Some others companies using [kpack](https://github.com/pivotal/kpack/) are: - -- [Bloomberg](http://techatbloomberg.com/opensource) -- [WPengine](https://wpengine.com/) -- [Terasky](https://www.terasky.com/) -- [SAP](https://www.sap.com/) - # Migration ### Repositories @@ -296,7 +285,8 @@ For each repository ### CI / CD Pipelines -[kpack's](https://github.com/pivotal/kpack/) CI/CD infrastructure currently runs on internal VMware infrastructure. [kpack's](https://github.com/pivotal/kpack/) CI/CD pipelines will need to be rebuilt on [CNB](https://buildpacks.io/) infrastructure. +[kpack's](https://github.com/pivotal/kpack/) CI/CD pipelines were rebuilt to use [github actions](https://github.com/pivotal/kpack/tree/main/.github). +In order for [kpack's](https://github.com/pivotal/kpack/) to run windows acceptance tests it requires a kubernetes cluster with windows nodes. The hardware requirements are specify in the following section. ##### Hardware requirements From 8cb9a0ced17270eca3616d1ea84600131525ed72 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 11 Apr 2023 14:50:54 -0500 Subject: [PATCH 108/222] updating the destination github repository to point to the buildpacks-community Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 8a20ee3f4..d9c0db67a 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -274,9 +274,9 @@ The following table shows the candidates repositories to be transferred | Origin Repo | Description | Owner | Destination Repo | Owner | | --- | --- | --- | --- | --- | -| [https://github.com/pivotal/kpack](https://github.com/pivotal/kpack) | kpack source code | Pivotal | [https://github.com/buildpacks/kpack](https://github.com/buildpacks/kpack) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | -| [https://github.com/vmware-tanzu/kpack-cli](https://github.com/vmware-tanzu/kpack-cli) | kpack CLI | VMware | [https://github.com/buildpacks/kpack-cli](https://github.com/buildpacks/kpack-cli) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | -| [https://github.com/vmware-tanzu/homebrew-kpack-cli](https://github.com/vmware-tanzu/homebrew-kpack-cli) | Homebrew tap for the kpack CLI | VMware | [https://github.com/buildpacks/homebrew-kpack-cli](https://github.com/buildpacks/homebrew-kpack-cli) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | +| [https://github.com/pivotal/kpack](https://github.com/pivotal/kpack) | kpack source code | Pivotal | [https://github.com/buildpacks-community/kpack](https://github.com/buildpacks/kpack) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | +| [https://github.com/vmware-tanzu/kpack-cli](https://github.com/vmware-tanzu/kpack-cli) | kpack CLI | VMware | [https://github.com/buildpacks-community/kpack-cli](https://github.com/buildpacks/kpack-cli) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | +| [https://github.com/vmware-tanzu/homebrew-kpack-cli](https://github.com/vmware-tanzu/homebrew-kpack-cli) | Homebrew tap for the kpack CLI | VMware | [https://github.com/buildpacks-community/homebrew-kpack-cli](https://github.com/buildpacks/homebrew-kpack-cli) | [CNB Technical Oversight Committee](https://github.com/buildpacks/community/blob/main/GOVERNANCE.md#technical-oversight-committee) | For each repository From 3bb9ae491604b3d03e14f0b53d597fa03b636a44 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 12 Apr 2023 10:28:13 -0500 Subject: [PATCH 109/222] adding link to the DEVELOPMENT.md pull request Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index d9c0db67a..1ea6e63a6 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -15,26 +15,26 @@ This RFC proposes the donation of the open-source project [kpack](https://github Following the process defined in the [Buildpack Commnity RFC](https://github.com/buildpacks/rfcs/blob/main/text/0117-buildpacks-community.md) the following table presents the criteria used to evaluate the project. -| Criteria | Evidence | -|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------| -| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | -| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | -| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | -| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149/files) | -| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | +| Criteria | Evidence | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| +| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | +| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | +| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | +| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | | The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. || | The list of project maintainers must be publicly available and controlled through a Github team. || -| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | -| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section | +| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | +| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section | | The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). || | There should have visible automated testing for all repositories that are part of the project. || | The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. || -| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149/files) | -| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. || -| The has a file - ADOPTERS.md: A list of adopters of the project. | See Adopters [Pull Request](https://github.com/pivotal/kpack/pull/1142) | -| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See Versioning [Pull Request](https://github.com/pivotal/kpack/pull/1150) | +| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See Development [Pull Request](https://github.com/pivotal/kpack/pull/1185) | +| The has a file - ADOPTERS.md: A list of adopters of the project. | See Adopters [Pull Request](https://github.com/pivotal/kpack/pull/1142) | +| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See Versioning [Pull Request](https://github.com/pivotal/kpack/pull/1150) | | The has a file - RELEASE.md: A guide to how releases are done for the project. || -| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149/files) | +| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | # Definitions From e27efb917dd00045f8d83f92c4ba8c1773d5e49e Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 13 Apr 2023 10:43:08 -0400 Subject: [PATCH 110/222] RFC 0120 [#281] Signed-off-by: Natalie Arellano --- text/{0000-cvebackports.md => 0120-cvebackports.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-cvebackports.md => 0120-cvebackports.md} (97%) diff --git a/text/0000-cvebackports.md b/text/0120-cvebackports.md similarity index 97% rename from text/0000-cvebackports.md rename to text/0120-cvebackports.md index ce02872bf..1d20a99ff 100644 --- a/text/0000-cvebackports.md +++ b/text/0120-cvebackports.md @@ -3,10 +3,10 @@ - Name: CVE discretional Patching - Start Date: 2023-03-08 - Author(s): joe-kimmel-vmw, natalieparellano -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#281](https://github.com/buildpacks/rfcs/pull/281) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: N/A - Supersedes: N/A # Summary From a4188254cea29cff989ec8f7c5aee4c255ec970e Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 19 Apr 2023 12:13:05 -0500 Subject: [PATCH 111/222] WIP - initial version Signed-off-by: Juan Bustamante --- text/0000-pack-manifest-list-commands.md | 148 +++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 text/0000-pack-manifest-list-commands.md diff --git a/text/0000-pack-manifest-list-commands.md b/text/0000-pack-manifest-list-commands.md new file mode 100644 index 000000000..bfe9db6c4 --- /dev/null +++ b/text/0000-pack-manifest-list-commands.md @@ -0,0 +1,148 @@ +# Meta +[meta]: #meta +- Name: Manifest List Commands for Pack +- Start Date: 2023-04-19 +- Author(s): [Juan Bustamante](Juan Bustamante) +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +The problem for adding support for multi-arch buildpacks can be divided into two parts: +- Support buildpack authors to **migrate their existing buildpacks** to support multi-arch . +- Support buildpack authors to **create new buildpacks and builders** that handle multi-arch from the beginning. + +This RFC proposes to create a new set of CRUD commands in pack to handle manifest lists, which will be used to support the first part of the problem. + +# Definitions +[definitions]: #definitions + +- Image Manifest: The image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. See [spec](https://github.com/opencontainers/image-spec/blob/main/manifest.md) +- Image Index: The image index is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. See [spec](https://github.com/opencontainers/image-spec/blob/main/image-index.md) + +# Motivation +[motivation]: #motivation + +- Why should we do this? + +The uses of ARM architecture in the cloud and edge computing has been growing rapidly. The CNCF community has been also growing in the last years, and there is a need to support multi-arch for all the projects. The buildpacks community is not an exception, issues like: +- [It would be nice to support easily creating a manifest list packages and builders](https://github.com/buildpacks/pack/issues/1460) +- [Provide a way to specify desired platform when creating packages and builders](https://github.com/buildpacks/pack/issues/1459) +- [Multi arch image build support](https://github.com/buildpacks/pack/issues/1570) + +Or the conversations around this topic in our [slack channel](https://cloud-native.slack.com/archives/C032LNSMY0P), even the [talk at Kubecon NA 2022](https://www.youtube.com/watch?v=Sdr5axlOnDI&list=PLj6h78yzYM2O5aNpRM71NQyx3WUe1xpTn&index=76) demonstrate the interest from the community in this feature. + +- What use cases does it support? + +Currently, buildpack authors can build and package their buildpacks for different OS and Architectures, but when they distribute them the URI for a buildpack can’t disambiguate, they need to use different tags to differentiate between them. This makes harder for users to consume those Buildpacks. +The solution is to share a single URI for all the different OS and Architectures, and the way to do that is using a manifest list. + +Adding commands to support the operations to handle the manifest list will allow buildpack authors to migrate their existing buildpacks to support multi-arch, without afecting their current existing process. + +- What is the expected outcome? + +The expected outcome is to have a set of commands in pack to handle manifest lists, for example: +- A command to create a manifest list from a set of images +- A command to push the manifest list to a registry +- A command to update or delete a manifest list + +# What it is +[what-it-is]: #what-it-is + +The proposal is to add a new command `pack manifest` and different subcommands. +- `pack manifest create` will create a local manifest list for annotating and pushing to a registry +- `pack manifest annotate` will add additional information like os, arch or variant to an existing local manifest list +- `pack manifest add` will add an image to an existing manifest list +- `pack manifest remove` will remove an image from a manifest list +- `pack manifest push` will push a manifest list to a registry + +Our target user affected by the feature is: Buildpack Authors + + + +# How it Works +[how-it-works]: #how-it-works + +This is the technical portion of the RFC, where you explain the design in sufficient detail. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. + +# Migration +[migration]: #migration + +This section should document breaks to public API and breaks in compatibility due to this RFC's proposed changes. In addition, it should document the proposed steps that one would need to take to work through these changes. Care should be give to include all applicable personas, such as platform developers, buildpack developers, buildpack users and consumers of buildpack images. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +We should decide to do not add this feature and users could use tools like `docker` or `podman` to create and handle their manifest list, however this is a poor user experience forcing users to use different tools to fulfill scenarios that are part of the business domain of pack. + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + +We also have in mind to improve existing commands like `pack builder create` and `pack buildpack package` to support the creation of a manifest list without the need of a new command. However, this approach could not be suitable for buildpack authors who are maintaining existing buildpacks and they will need to change their current process to generate multi-arch images. + +- Why is this proposal the best? + +Because we will provide the tool to our end users to solve their problems without the need of using other tools. + +- What is the impact of not doing this? + +The impact of not doing this is that users will need to use other tools to create and handle their manifest list, which is a poor user experience. + +# Prior Art +[prior-art]: #prior-art + +These features are inspired in similar commands in other tools like: +- [docker](https://docs.docker.com/engine/reference/commandline/manifest/) +- [podman](https://docs.podman.io/en/v3.2.0/markdown/podman-manifest-create.1.html) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? +- What parts of the design do you expect to be resolved through implementation of the feature? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes +Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. +Examples of a spec. change might be new lifecycle flags, new `buildpack.toml` fields, new fields in the buildpackage label, etc. +This section is not intended to be binding, but as discussion of an RFC unfolds, if spec changes are necessary, they should be documented here. + +# History +[history]: #history + + From 2878868f8df805256804bad43c3cc4ab1250fba0 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 20 Apr 2023 08:11:36 -0500 Subject: [PATCH 112/222] Update text/0000-pack-manifest-list-commands.md Co-authored-by: Aidan Delaney Signed-off-by: Juan Bustamante --- text/0000-pack-manifest-list-commands.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-pack-manifest-list-commands.md b/text/0000-pack-manifest-list-commands.md index bfe9db6c4..2d96778cc 100644 --- a/text/0000-pack-manifest-list-commands.md +++ b/text/0000-pack-manifest-list-commands.md @@ -53,7 +53,7 @@ The expected outcome is to have a set of commands in pack to handle manifest lis # What it is [what-it-is]: #what-it-is -The proposal is to add a new command `pack manifest` and different subcommands. +The proposal is to add a new _experimental_ command `pack manifest` and different subcommands. The `pack manifest` commands will initially be gated behind `pack config experimental`. The `pack manifest` commands will move from experimental status to supported status when maintainers deem it appropriate. - `pack manifest create` will create a local manifest list for annotating and pushing to a registry - `pack manifest annotate` will add additional information like os, arch or variant to an existing local manifest list - `pack manifest add` will add an image to an existing manifest list From ae208e288bd3fffaa90ddc11638027c1193fa31a Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 20 Apr 2023 14:10:37 -0500 Subject: [PATCH 113/222] Adding what is this section Signed-off-by: Juan Bustamante --- text/0000-pack-manifest-list-commands.md | 88 +++++++++++++++++++++++- 1 file changed, 87 insertions(+), 1 deletion(-) diff --git a/text/0000-pack-manifest-list-commands.md b/text/0000-pack-manifest-list-commands.md index 2d96778cc..cb8507d66 100644 --- a/text/0000-pack-manifest-list-commands.md +++ b/text/0000-pack-manifest-list-commands.md @@ -60,7 +60,93 @@ The proposal is to add a new _experimental_ command `pack manifest` and differen - `pack manifest remove` will remove an image from a manifest list - `pack manifest push` will push a manifest list to a registry -Our target user affected by the feature is: Buildpack Authors +Our target user affected by the feature is: **Buildpack Authors**. Let's see some examples of how this feature will work. + +Currently, if we check [sample-packages](https://hub.docker.com/r/cnbs/sample-package/tags) at dockerhub we will notice that we have a composed buildpack called `hello-universe` and we offer two tags to support different architectures: +- `cnbs/sample-package:hello-universe` for linux and +- `cnbs/sample-package:hello-universe-windows`. + +Maybe our linux version should be called `cnbs/sample-package:hello-universe-linux` to keep the same naming convention, but we will keep it as it is for simplicity. If we want to distribute the `hello-universe` buildpack for any **architecture/os/variant** combination we need to use a tool outside the CNB ecosystem to create a manifest list. But, with our proposal experimental commands on pack we can do: + +```bash +$ pack manifest create cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe \ + cnbs/sample-package:hello-universe-windows +``` + +By default the command will create a manifest list in the local daemon using the docker media types [Version 2 schema 2](https://docs.docker.com/registry/spec/manifest-v2-2/) with a content similar to: + +```json +{ + "schemaVersion": 2, + "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", + "manifests": [ + { + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "size": 226083, + "digest": "sha256: 87a832fd6a8d6995d336c740eb6f3da015401a6e564fcbe95ee1bf37557a8225", + "platform": { + "os": "linux", + "architecture": "" + } + }, + { + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "size": 226083, + "digest": "sha256:670d62fbee841d256a706801a03be9c84d37fc2cd6ef7538a7af9985c3d2ed8b", + "platform": { + "os": "windows", + "architecture": "" + } + } + ] +} +``` +The idea to save the manifest list locally is to allow the user to update the manifest before pushing it to a registry, +in this case, we need to define the **architecture** field because it is empty in our examples. + +We can use the `pack manifest annotate` command to add the architecture information: + +```bash +$ pack manifest annotate --arch amd64 cnbs/sample-package:hello-multiarch-universe cnbs/sample-package:hello-universe +$ pack manifest annotate --arch amd64 cnbs/sample-package:hello-multiarch-universe cnbs/sample-package:hello-universe-windows +``` + +After executing these commands, our local manifest list will be updated as follows: + +```json +{ + "schemaVersion": 2, + "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", + "manifests": [ + { + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "size": 940, + "digest": "sha256:87a832fd6a8d6995d336c740eb6f3da015401a6e564fcbe95ee1bf37557a8225", + "platform": { + "architecture": "amd64", + "os": "linux" + } + }, + { + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "size": 1148, + "digest": "sha256:670d62fbee841d256a706801a03be9c84d37fc2cd6ef7538a7af9985c3d2ed8b", + "platform": { + "architecture": "amd64", + "os": "windows" + } + } + ] +} +``` + +Finally, our manifest list is ready to be pushed to a registry, and we can use the `pack manifest push` command to do it: + +```bash +$ pack manifest push cnbs/sample-package:hello-multiarch-universe +``` +And our manifest list should be published at [dockerhub](https://hub.docker.com/r/cnbs/sample-package/tags) as `cnbs/sample-package:hello-multiarch-universe`, asuming that we have the proper credentials to push the image. - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) From 3086c836389231984046d47238d2ccc14ceb2b04 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Fri, 28 Apr 2023 15:48:36 +0100 Subject: [PATCH 115/222] Fix example process definition in RFC 0093 RFC 0093 contains a number of examples of how to migrate a buildpack away from `direct=false` mode. One of these examples was missing the "bash" command prior to the "-c" argument. Signed-off-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0093-remove-shell-processes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0093-remove-shell-processes.md b/text/0093-remove-shell-processes.md index 269fee7c8..9a7f1876f 100644 --- a/text/0093-remove-shell-processes.md +++ b/text/0093-remove-shell-processes.md @@ -108,7 +108,7 @@ Using the new API this process could look like: ``` [[processes]] type = "bash" -command = ["-c", "dotnet", "my-app.dll", "--urls", "http://0.0.0.0:${PORT:-8080}"] +command = ["bash", "-c", "dotnet", "my-app.dll", "--urls", "http://0.0.0.0:${PORT:-8080}"] default = true ``` Things to note: @@ -400,4 +400,4 @@ In addition to the changes described originally in 0093 we'd like some way of ve Why was this amendment necessary? -The RFC text should reflect what was actually implemented / agreed upon to avoid confusion. \ No newline at end of file +The RFC text should reflect what was actually implemented / agreed upon to avoid confusion. From 8c8d7574ca324e73702de8132714f34c347de6f7 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 17 May 2023 17:40:59 -0500 Subject: [PATCH 116/222] updating latest changes for merged PR and removing Nicholas Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 33 +++++++++++++++--------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 1ea6e63a6..220f0de5b 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -15,26 +15,26 @@ This RFC proposes the donation of the open-source project [kpack](https://github Following the process defined in the [Buildpack Commnity RFC](https://github.com/buildpacks/rfcs/blob/main/text/0117-buildpacks-community.md) the following table presents the criteria used to evaluate the project. -| Criteria | Evidence | -|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| -| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | -| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | -| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | -| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | -| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | +| Criteria | Evidence | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| +| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | +| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | +| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | +| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | | The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. || | The list of project maintainers must be publicly available and controlled through a Github team. || -| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | -| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section | -| The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). || -| There should have visible automated testing for all repositories that are part of the project. || +| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | +| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section and [People](https://github.com/orgs/buildpacks-community/people) in CNB community organization | +| The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). | See [issues](https://github.com/pivotal/kpack/issues) and [pull requests](https://github.com/pivotal/kpack/pulls) | +| There should have visible automated testing for all repositories that are part of the project. | See [codecov](https://app.codecov.io/gh/pivotal/kpack) | | The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. || -| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149) | -| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See Development [Pull Request](https://github.com/pivotal/kpack/pull/1185) | -| The has a file - ADOPTERS.md: A list of adopters of the project. | See Adopters [Pull Request](https://github.com/pivotal/kpack/pull/1142) | -| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See Versioning [Pull Request](https://github.com/pivotal/kpack/pull/1150) | +| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | +| The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | +| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | | The has a file - RELEASE.md: A guide to how releases are done for the project. || -| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | # Definitions @@ -328,7 +328,6 @@ The [kpack's](https://github.com/pivotal/kpack/) maintainers that will be nomina |------------------|----------------------------------------------------|--------------| | Matthew McNew | [@matthewmcnew](https://github.com/matthewmcnew) | VMware | | Tom Kennedy | [@tomkennedy513](https://github.com/tomkennedy513) | VMware | -| Nicholas Carlson | [@ncarlson](https://github.com/ncarlson) | VMWare | | Yael Harel | [@yaelharel](https://github.com/yaelharel) | VMware | | Daniel Chen | [@chenbh](https://github.com/chenbh) | VMware | | Juan Bustamante | [@jjbustamante](https://github.com/jjbustamante) | VMware | From fa6df69fd813adbc671701e717e9ddd3b206d15d Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Thu, 18 May 2023 09:04:47 +0100 Subject: [PATCH 117/222] Update text/0000-kpack-donation-to-cnb.md Co-authored-by: Terence Lee Signed-off-by: Sambhav Kothari --- text/0000-kpack-donation-to-cnb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 220f0de5b..23c1d2a9f 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -11,7 +11,7 @@ # Summary -This RFC proposes the donation of the open-source project [kpack](https://github.com/pivotal/kpack/) into the [Cloud Native Community Organization Buildpack](https://github.com/buildpacks-community). +This RFC proposes the donation of the open-source project [kpack](https://github.com/pivotal/kpack/) into the [Cloud Native Buildpacks Community Organization](https://github.com/buildpacks-community) as a vendor neutral staging ground under the CNB governance umbrella. Once the project is deemed sufficiently mature, the project will be moved under the [Cloud Native Buildpacks Organization](https://github.com/buildpacks). Following the process defined in the [Buildpack Commnity RFC](https://github.com/buildpacks/rfcs/blob/main/text/0117-buildpacks-community.md) the following table presents the criteria used to evaluate the project. From ef5f340da20cc7be31712ca58c7a6abfbd51ca5f Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 18 May 2023 14:38:23 -0500 Subject: [PATCH 118/222] Adding reference to Release.md PR Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 23c1d2a9f..68565716d 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -33,7 +33,7 @@ Following the process defined in the [Buildpack Commnity RFC](https://github.com | The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | | The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | | The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | -| The has a file - RELEASE.md: A guide to how releases are done for the project. || +| The has a file - RELEASE.md: A guide to how releases are done for the project. | See Release [Pull Request](https://github.com/pivotal/kpack/pull/1216) | | The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | # Definitions From 2f85dba546c463a04f9d57e9af76abc8cdd2b741 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 18 May 2023 15:35:52 -0500 Subject: [PATCH 119/222] adding go-licenses report link Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 35 +++++++++++++++--------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 68565716d..4e2924627 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -15,26 +15,27 @@ This RFC proposes the donation of the open-source project [kpack](https://github Following the process defined in the [Buildpack Commnity RFC](https://github.com/buildpacks/rfcs/blob/main/text/0117-buildpacks-community.md) the following table presents the criteria used to evaluate the project. -| Criteria | Evidence | -|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| -| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | -| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | -| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | -| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | -| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | +| Criteria | Evidence | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| +| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | +| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | + | List of all external dependencies with licensing info and they’re permissively licensed with a Apache 2.0 compatible license | See [go-licenses](https://github.com/google/go-licenses) [report](https://drive.google.com/file/d/1SvPdl69Lhn0TTx_TaxcesfajuYY9uiNd/view?usp=share_link) | +| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | +| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | | The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. || | The list of project maintainers must be publicly available and controlled through a Github team. || -| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | -| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section and [People](https://github.com/orgs/buildpacks-community/people) in CNB community organization | -| The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). | See [issues](https://github.com/pivotal/kpack/issues) and [pull requests](https://github.com/pivotal/kpack/pulls) | -| There should have visible automated testing for all repositories that are part of the project. | See [codecov](https://app.codecov.io/gh/pivotal/kpack) | +| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | +| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section and [People](https://github.com/orgs/buildpacks-community/people) in CNB community organization | +| The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). | See [issues](https://github.com/pivotal/kpack/issues) and [pull requests](https://github.com/pivotal/kpack/pulls) | +| There should have visible automated testing for all repositories that are part of the project. | See [codecov](https://app.codecov.io/gh/pivotal/kpack) | | The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. || -| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149) | -| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | -| The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | -| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | -| The has a file - RELEASE.md: A guide to how releases are done for the project. | See Release [Pull Request](https://github.com/pivotal/kpack/pull/1216) | -| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | +| The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | +| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | +| The has a file - RELEASE.md: A guide to how releases are done for the project. | See Release [Pull Request](https://github.com/pivotal/kpack/pull/1216) | +| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | # Definitions From b0a985efa084fb622db46cdbecbcdbe81f769d12 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 18 May 2023 15:37:48 -0500 Subject: [PATCH 120/222] adding go-licenses report link Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 36 +++++++++++++++--------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 4e2924627..9d355d771 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -15,27 +15,27 @@ This RFC proposes the donation of the open-source project [kpack](https://github Following the process defined in the [Buildpack Commnity RFC](https://github.com/buildpacks/rfcs/blob/main/text/0117-buildpacks-community.md) the following table presents the criteria used to evaluate the project. -| Criteria | Evidence | -|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| -| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | -| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | - | List of all external dependencies with licensing info and they’re permissively licensed with a Apache 2.0 compatible license | See [go-licenses](https://github.com/google/go-licenses) [report](https://drive.google.com/file/d/1SvPdl69Lhn0TTx_TaxcesfajuYY9uiNd/view?usp=share_link) | -| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | -| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | -| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | +| Criteria | Evidence | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | +| The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | + | List of all external dependencies with licensing info and they’re permissively licensed with a Apache 2.0 compatible license | See [report](https://drive.google.com/file/d/1SvPdl69Lhn0TTx_TaxcesfajuYY9uiNd/view?usp=share_link) generated using [go-licenses](https://github.com/google/go-licenses) | +| It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | +| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | | The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. || | The list of project maintainers must be publicly available and controlled through a Github team. || -| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | -| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section and [People](https://github.com/orgs/buildpacks-community/people) in CNB community organization | -| The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). | See [issues](https://github.com/pivotal/kpack/issues) and [pull requests](https://github.com/pivotal/kpack/pulls) | -| There should have visible automated testing for all repositories that are part of the project. | See [codecov](https://app.codecov.io/gh/pivotal/kpack) | +| The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | +| All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section and [People](https://github.com/orgs/buildpacks-community/people) in CNB community organization | +| The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). | See [issues](https://github.com/pivotal/kpack/issues) and [pull requests](https://github.com/pivotal/kpack/pulls) | +| There should have visible automated testing for all repositories that are part of the project. | See [codecov](https://app.codecov.io/gh/pivotal/kpack) | | The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. || -| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149) | -| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | -| The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | -| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | -| The has a file - RELEASE.md: A guide to how releases are done for the project. | See Release [Pull Request](https://github.com/pivotal/kpack/pull/1216) | -| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | +| The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | +| The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | +| The has a file - RELEASE.md: A guide to how releases are done for the project. | See Release [Pull Request](https://github.com/pivotal/kpack/pull/1216) | +| The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | # Definitions From 7840070df2a390b387cf2849cb09374f12fb7815 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 23 May 2023 15:50:53 -0500 Subject: [PATCH 121/222] updating some links to require documents Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 9d355d771..f85501fd3 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -23,14 +23,14 @@ Following the process defined in the [Buildpack Commnity RFC](https://github.com | It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | | The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | | The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | -| The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. || +| The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. | See Governance [Pull Request](https://github.com/pivotal/kpack/pull/1220) | | The list of project maintainers must be publicly available and controlled through a Github team. || | The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | | All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section and [People](https://github.com/orgs/buildpacks-community/people) in CNB community organization | | The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). | See [issues](https://github.com/pivotal/kpack/issues) and [pull requests](https://github.com/pivotal/kpack/pulls) | | There should have visible automated testing for all repositories that are part of the project. | See [codecov](https://app.codecov.io/gh/pivotal/kpack) | | The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. || -| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See Contributing [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See [Contributing](https://github.com/pivotal/kpack/blob/main/CONTRIBUTING.md) | | The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | | The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | | The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | From 30ce074e82fa2626135af7b496c0ab65092478c5 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 24 May 2023 09:29:50 -0500 Subject: [PATCH 122/222] updating link to RELEASE.md Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index f85501fd3..7e4ced6a5 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -34,7 +34,7 @@ Following the process defined in the [Buildpack Commnity RFC](https://github.com | The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | | The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | | The has a file - VERSIONING.md: A guide to how versioning is done for the project. | See [Versioning](https://github.com/pivotal/kpack/blob/main/VERSIONING.md) | -| The has a file - RELEASE.md: A guide to how releases are done for the project. | See Release [Pull Request](https://github.com/pivotal/kpack/pull/1216) | +| The has a file - RELEASE.md: A guide to how releases are done for the project. | See [Release](https://github.com/pivotal/kpack/blob/main/RELEASE.md) | | The has a file - SECURITY.md: A guide to how security vulnerabilities should be reported. | See Security [Pull Request](https://github.com/pivotal/kpack/pull/1149) | # Definitions @@ -329,11 +329,10 @@ The [kpack's](https://github.com/pivotal/kpack/) maintainers that will be nomina |------------------|----------------------------------------------------|--------------| | Matthew McNew | [@matthewmcnew](https://github.com/matthewmcnew) | VMware | | Tom Kennedy | [@tomkennedy513](https://github.com/tomkennedy513) | VMware | -| Yael Harel | [@yaelharel](https://github.com/yaelharel) | VMware | | Daniel Chen | [@chenbh](https://github.com/chenbh) | VMware | | Juan Bustamante | [@jjbustamante](https://github.com/jjbustamante) | VMware | -Also those members are willing to become more involved with CNB projects and become **Platform maintainers** in the near future. +Also, those members are willing to become more involved with CNB projects and become **Platform maintainers** in the near future. Outside VMware, the following contributors manifested their desired to become [kpack's](https://github.com/pivotal/kpack/) **component maintainer**. From 9ff4797e8091288afa916a7fee2eb1332d398819 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Fri, 9 Jun 2023 15:44:53 -0700 Subject: [PATCH 123/222] RFC for 2023H2 Roadmap Signed-off-by: Terence Lee --- text/0000-2023H2-roadmap.md | 171 ++++++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 text/0000-2023H2-roadmap.md diff --git a/text/0000-2023H2-roadmap.md b/text/0000-2023H2-roadmap.md new file mode 100644 index 000000000..5516fbe45 --- /dev/null +++ b/text/0000-2023H2-roadmap.md @@ -0,0 +1,171 @@ +# Meta +[meta]: #meta +- Name: 2023H2 Roadmap +- Start Date: 2023-06-07 +- Author(s): hone +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +This RFC details the second half of the 2023 Roadmap leading up to KubeCon NA. + +# Definitions +[definitions]: #definitions + +## Readmap Status Items +* Finished - The item has already been completed and there is no need to put it on the current roadmap. +* Continue - More work is needed and it will be continued as part of the current roadmap. +* Defer - This item will be parked for now and doesn't make the cut for the current roadmap. + +# Motivation +[motivation]: #motivation + +With KubeCon NA coming up around the corner, it's time to plan out what we want to achieve as a project. It's also an opportunity to review items from the H1 roadmap. + +# What it is +[what-it-is]: #what-it-is + +## 2023H2 Roadmap + +This roadmap is going to be split into two sections covering the first the H1 Roadmap and status, as well as new items we want to tackle leading up to KubeCon NA. + +### Items from [2023H1] +[items-from-2023h1]: #items-from-2023h1 + +As a project, we've made good progress on our H1 roadmap, but still need some more time to get some of them across the finish line. + +#### Release Base Image Extension +* Owner: @natalieparellano +* Status: Continue +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0105-dockerfiles.md) + +This is close and will be released as part of Platform `0.12`, Buildpack `0.10`, lifecycle `0.17.0`, and pack `0.30.0`. + +#### Remove Stacks & Mixins +* Owner: @jkutner +* Status: Continue +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) + +This is close and will be released as part of Platform `0.12`, Buildpack `0.10`, lifecycle `0.17.0`, and pack `0.30.0`. + +#### Execution Environments RFC +* Owner: @hone +* Status: Continue +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/274) + +The RFC is written, but feedback needs to be incoporated before re-opening for review. + +#### Project Health +* Owner: @samj1912 +* Status: Finished +* Links: [Buildpacks Community Organization RFC](https://github.com/buildpacks/rfcs/pull/273) + +The RFC has been merged and the buildpacks community GitHub org have been created. + +#### Pack Test Cleaning/Optimizations +* Owner: @dfreilich +* Status: Defer +* Links: [Pack Pull Request](https://github.com/buildpacks/pack/pull/1498) + +This item has been deferred for now, but we will work with anyone who wants to push this forward. + +### New Items + +#### Community Engagement Health Checks +* Owner: @microwavables (Team Lead sponsor @jkutner) +* Links: [VMware Tanzu Community Engagement Health Checks](https://github.com/vmware-tanzu/community-engagement/blob/main/HEALTHCHECKS.md) + +The project is lucky to have a Community Manager! This is one of the projects proposed by @microwavables to set a base line to measure how we're doing as a community. + +#### RFC for Buildpack Author Observability +* Owner: @joshwlewis (Team Lead sponsor @hone) +* Links: TBD + +Currently, Buildpack Authors have little to no tools around visibility with their buildpacks as they run on a platform, including `pack`. In some of the Heroku v2 buildpacks, they implemented logging that could be handed off to the platform by running `bin/report`. This work stream is about standardizing output for both successful AND failed builds that Buildpack Authors can use to instrument their buildpack. + +#### Private Registry Mirrors +* Owner: @jabrown85 +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/285) + +A platform operator can configure registry mirrors that lifecycle could use without needing for the manifest to have to point to it. This will allow a platform to reduce the risk of service operations from external registry sources and reduce public network bandwidth usage. The resulting image when taken off platform will also function without needing access to the registry mirror. + +#### kpack Donation +* Owner: @jjbustamante (Team Lead sponsor @hone) +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/235) + +`kpack` is being proposed to be donated as an open source project in the Cloud Native Buildpacks' new Community Organization as a vendor neutral staging ground. This will give the project space to grow the project contributor base from multiple vendors. While work has started on this in H1, this item represents our commitment as a project to see this through and set this project up for success under the CNB governance umbrella. + +#### Cosign Integration / OCI Image Manifest v1.1 +* Owner: @natalieparellano +* Links: [Cosign Integration RFC](https://github.com/buildpacks/rfcs/pull/195), [SBOM layer RFC](https://github.com/buildpacks/rfcs/pull/278), + +While CNBs support SBOMs today, they were designed a few years ago and tooling around them have been evolving. This work stream is about making CNBs integrate better with tools like [cosign's SBOM spec](https://github.com/sigstore/cosign/blob/main/specs/SBOM_SPEC.md) and the upcoming [OCI References](https://github.com/opencontainers/image-spec/issues/827) feature in OCI Image Manifest 1.1. + +#### Pack OCI Manifest Support +* Owner: @jjbustamante +* Links: [RFC](https://github.com/buildpacks/rfcs/pull/283) + +Multi-arch support has been a highly request feature with the growing popularity of the ARM architecture. In order to better support this with Buildpacks, the first step will be to able to use manifest lists to provide a single URI for Buildpacks that support multiple architectures. + +# How it Works +[how-it-works]: #how-it-works + +See [What it is](#what-it-is) for the details. We'll be following the same process from the [2023H1 Roadmap RFC](https://github.com/buildpacks/rfcs/blob/main/text/0118-2023H1-roadmap.md) if approved. + +# Migration +[migration]: #migration + +N/A + +# Drawbacks +[drawbacks]: #drawbacks + +- Agreeing to more work, while we still need to finish 2023H1 items. + +# Alternatives +[alternatives]: #alternatives + +- Do Nothing and just hunker down on our existing items. + +# Prior Art +[prior-art]: #prior-art + +See [Prior Art from 2023H1 Roadmap RFC](https://github.com/buildpacks/rfcs/blob/main/text/0118-2023H1-roadmap.md#prior-art). + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? +- What parts of the design do you expect to be resolved through implementation of the feature? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +N/A + +# History +[history]: #history + + From 3741035bae6feb255152e61ab2811d91f41cb667 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 22 Jun 2023 07:59:50 -0500 Subject: [PATCH 124/222] Updating the links to the governance.md file recently merged in kpack Signed-off-by: Juan Bustamante --- text/0000-kpack-donation-to-cnb.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0000-kpack-donation-to-cnb.md index 7e4ced6a5..b708bc876 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0000-kpack-donation-to-cnb.md @@ -19,17 +19,17 @@ Following the process defined in the [Buildpack Commnity RFC](https://github.com |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | The project must be a tooling, platform or integration that is related to Cloud Native Buildpacks. | See [Motivation](#motivation) section | | The project must be open source and licensed under Apache 2.0. | See [License](https://github.com/pivotal/kpack/blob/main/LICENSE) | - | List of all external dependencies with licensing info and they’re permissively licensed with a Apache 2.0 compatible license | See [report](https://drive.google.com/file/d/1SvPdl69Lhn0TTx_TaxcesfajuYY9uiNd/view?usp=share_link) generated using [go-licenses](https://github.com/google/go-licenses) | +| List of all external dependencies with licensing info and they’re permissively licensed with a Apache 2.0 compatible license | See [report](https://drive.google.com/file/d/1SvPdl69Lhn0TTx_TaxcesfajuYY9uiNd/view?usp=share_link) generated using [go-licenses](https://github.com/google/go-licenses) | | It must follow the Cloud Native Computing Foundation Code of Conduct. | See [Code of conduct](https://github.com/pivotal/kpack/blob/main/CODE_OF_CONDUCT.md) | -| The project must enable DCO signoff for all commits. | See Sign-off process [Pull Request](https://github.com/pivotal/kpack/pull/1149) | +| The project must enable DCO signoff for all commits. | See [Sign-off process](https://github.com/pivotal/kpack/blob/main/CONTRIBUTING.md#sign-off-process) | | The project must be open to contributions and have a public issue tracker. | See public [issue tracker](https://github.com/pivotal/kpack/issues) | -| The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. | See Governance [Pull Request](https://github.com/pivotal/kpack/pull/1220) | -| The list of project maintainers must be publicly available and controlled through a Github team. || +| The project must have a governance document that clearly defines the project maintainers and how they are elected. Each project may choose to define their own governance model as long as it is clearly documented and allows for project maintainers to be elected from the community. | See [Governance](https://github.com/pivotal/kpack/blob/main/GOVERNANCE.md) | +| The list of project maintainers must be publicly available and controlled through a Github team. | See [Maintainers](https://github.com/pivotal/kpack/blob/main/MAINTAINERS.md) | | The project must use a CODEOWNERS file to define the maintainers for each repository. The CODEOWNERS file should reference the Github team that controls the list of maintainers. | See [CODEOWNERS](https://github.com/pivotal/kpack/blob/main/CODEOWNERS) file | | All project contributors must be members of the Buildpacks community organization. | See [Team Roles](#team-roles) section and [People](https://github.com/orgs/buildpacks-community/people) in CNB community organization | | The project must be actively maintained (i.e. issues and pull requests must be addressed regularly, approved pull requests must be merged or updated in a timely manner, etc.). | See [issues](https://github.com/pivotal/kpack/issues) and [pull requests](https://github.com/pivotal/kpack/pulls) | | There should have visible automated testing for all repositories that are part of the project. | See [codecov](https://app.codecov.io/gh/pivotal/kpack) | -| The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. || +| The project maintainers must conform to a set of best effort SLOs around patching critical CVEs when applicable to the project. | | | The has a file - CONTRIBUTING.md: A guide to how contributors should submit patches and the expectations around code review. | See [Contributing](https://github.com/pivotal/kpack/blob/main/CONTRIBUTING.md) | | The has a file - DEVELOPMENT.md: A guide to how contributors should develop the project. | See [Development](https://github.com/pivotal/kpack/blob/main/DEVELOPMENT.md) | | The has a file - ADOPTERS.md: A list of adopters of the project. | See [Adopters](https://github.com/pivotal/kpack/blob/main/ADOPTERS.md) | From bfb4778d077e3bda49d4dd88bc7dc471e6b042c4 Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Wed, 28 Jun 2023 11:31:04 -0500 Subject: [PATCH 125/222] Update 0115-rebase-immutable-image-ref.md Signed-off-by: Jesse Brown --- text/0115-rebase-immutable-image-ref.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0115-rebase-immutable-image-ref.md b/text/0115-rebase-immutable-image-ref.md index 02d548cb7..901911b49 100644 --- a/text/0115-rebase-immutable-image-ref.md +++ b/text/0115-rebase-immutable-image-ref.md @@ -3,9 +3,9 @@ - Name: Rebase by Image Digest Reference - Start Date: 2022-12-08 - Author(s): [@joeybrown-sf](https://github.com/joeybrown-sf) -- Status: Approved +- Status: Implemented - RFC Pull Request: [rfcs#262](https://github.com/buildpacks/rfcs/pull/262) -- CNB Pull Request: (leave blank) +- CNB Pull Request: https://github.com/buildpacks/lifecycle/pull/985 - CNB Issue: [buildpacks/lifecycle#983](https://github.com/buildpacks/lifecycle/issues/983) - Supersedes: N/A @@ -115,4 +115,4 @@ A brief description of the changes. ### Motivation Why was this amendment necessary? ----> \ No newline at end of file +---> From 6d37bbcde404abd83fcd89f9b91e747a90d6c76f Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 29 Jun 2023 08:53:27 -0700 Subject: [PATCH 126/222] add Export OCI Layout to the roadmap Signed-off-by: Terence Lee --- text/0000-2023H2-roadmap.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/text/0000-2023H2-roadmap.md b/text/0000-2023H2-roadmap.md index 5516fbe45..a3f87e190 100644 --- a/text/0000-2023H2-roadmap.md +++ b/text/0000-2023H2-roadmap.md @@ -107,11 +107,17 @@ A platform operator can configure registry mirrors that lifecycle could use with While CNBs support SBOMs today, they were designed a few years ago and tooling around them have been evolving. This work stream is about making CNBs integrate better with tools like [cosign's SBOM spec](https://github.com/sigstore/cosign/blob/main/specs/SBOM_SPEC.md) and the upcoming [OCI References](https://github.com/opencontainers/image-spec/issues/827) feature in OCI Image Manifest 1.1. #### Pack OCI Manifest Support -* Owner: @jjbustamante +* Owner: @jjbustamante (Team Lead sponsor @hone) * Links: [RFC](https://github.com/buildpacks/rfcs/pull/283) Multi-arch support has been a highly request feature with the growing popularity of the ARM architecture. In order to better support this with Buildpacks, the first step will be to able to use manifest lists to provide a single URI for Buildpacks that support multiple architectures. +#### Export to OCI Layout +* Owner: @jjbustamante (Team Lead sponsor @natalieparellano) +* Links: [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0119-export-to-oci.md) + +The RFC has been merged and the implementation is expected to be released in experimental mode on pack *v0.30.0* + # How it Works [how-it-works]: #how-it-works From 174ce566b5ba7eeb2406351f5024962e01c07cd8 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 4 Jul 2023 15:51:12 -0500 Subject: [PATCH 127/222] adding details to the section how it works Signed-off-by: Juan Bustamante --- text/0000-pack-manifest-list-commands.md | 47 +++++++++++++++++++++--- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/text/0000-pack-manifest-list-commands.md b/text/0000-pack-manifest-list-commands.md index b649e08d3..f8aa64a72 100644 --- a/text/0000-pack-manifest-list-commands.md +++ b/text/0000-pack-manifest-list-commands.md @@ -57,8 +57,10 @@ The proposal is to add a new _experimental_ command `pack manifest` and differen - `pack manifest create` will create a local manifest list for annotating and pushing to a registry - `pack manifest annotate` will add additional information like os, arch or variant to an existing local manifest list - `pack manifest add` will add an image to an existing manifest list -- `pack manifest remove` will remove an image from a manifest list +- `pack manifest delete` will delete a manifest list from local storage +- `pack manifest rm` will remove an image from a manifest list - `pack manifest push` will push a manifest list to a registry +- `pack manifest inspect` will show the manifest information stored in local storage Our target user affected by the feature is: **Buildpack Authors**. Let's see some examples of how this feature will work. @@ -66,7 +68,7 @@ Currently, if we check [sample-packages](https://hub.docker.com/r/cnbs/sample-pa - `cnbs/sample-package:hello-universe` for linux and - `cnbs/sample-package:hello-universe-windows`. -Maybe our linux version should be called `cnbs/sample-package:hello-universe-linux` to keep the same naming convention, but we will keep it as it is for simplicity. If we want to distribute the `hello-universe` buildpack for any **architecture/os/variant** combination we need to use a tool outside the CNB ecosystem to create a manifest list. But, with our proposal experimental commands on pack we can do: +Let's suppose our linux version is called `cnbs/sample-package:hello-universe-linux` to keep the same naming convention, but we will keep it as it is for simplicity. If we want to distribute the `hello-universe` buildpack for any **architecture/os/variant** combination we need to use a tool outside the CNB ecosystem to create a manifest list. With the proposed experimental commands on pack we can do: ```bash $ pack manifest create cnbs/sample-package:hello-multiarch-universe \ @@ -74,7 +76,7 @@ $ pack manifest create cnbs/sample-package:hello-multiarch-universe \ cnbs/sample-package:hello-universe-windows ``` -By default the command will create a manifest list in the local daemon using the docker media types [Version 2 schema 2](https://docs.docker.com/registry/spec/manifest-v2-2/) with a content similar to: +By default, the command will create a manifest list in the local storage using the docker media types [Version 2 schema 2](https://docs.docker.com/registry/spec/manifest-v2-2/) with a content similar to: ```json { @@ -161,9 +163,44 @@ This provides a high level overview of the feature. # How it Works [how-it-works]: #how-it-works -This is the technical portion of the RFC, where you explain the design in sufficient detail. +The proposal is to implement an abstraction of an OCI *Image Index* and expose it to users through `pack manifest` commands. -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +## Image Index Abstraction + +```mermaid +classDiagram + ManifestList <|-- remote_ManifestList + ManifestList <|-- local_ManifestList + + class remote_ManifestList { + +NewManifestList(repoName string, keychain authn.Keychain) ManifestList + } + + class local_ManifestList { + +NewManifestList(repoName string, path string) ManifestList + } + + class ManifestList { + <> + +Add(repoName string) error + +Remove(repoName string) error + +Save() error + } +``` + +Two implementations: *remote* and *local* are propose, *remote* will take care of implementing the interaction with an OCI registry and *local* will deal with the local storage operations. + +### Component Diagram + +Using a [C4 component diagram](https://c4model.com/#ComponentDiagram), we can define the high level interaction on pack. This design follows the same pattern for each command already implemented. + +![](https://hackmd.io/_uploads/B1PpJ-fKh.png) + +- *Image Factory*: is responsible for instantiate the *Image Index* abstraction based on the configuration require, it could be a *remote* or a *local* implementation +- *Image Index*: is the abstraction defined which exposes the operation methods we want to offer to users + - As we can see, depending on the implementation it will interact with the file system or with a remote registry + +### Considerations # Migration [migration]: #migration From 261778bf271e228b906ec00eb8932c350e2dd1f7 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 4 Jul 2023 17:23:27 -0500 Subject: [PATCH 128/222] adding details to the section how it works Signed-off-by: Juan Bustamante --- text/0000-pack-manifest-list-commands.md | 38 ++++++++++++++++++------ 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/text/0000-pack-manifest-list-commands.md b/text/0000-pack-manifest-list-commands.md index f8aa64a72..c1b709810 100644 --- a/text/0000-pack-manifest-list-commands.md +++ b/text/0000-pack-manifest-list-commands.md @@ -167,10 +167,19 @@ The proposal is to implement an abstraction of an OCI *Image Index* and expose i ## Image Index Abstraction +A new high level abstraction to represent an OCI Image Index is proposed, similar to the [Image](https://github.com/buildpacks/imgutil/blob/main/image.go) interface exposed in *imgutil* repository, +we proposed a new *ManifestList* interface to expose the behavior of an OCI Image Index. + ```mermaid classDiagram - ManifestList <|-- remote_ManifestList - ManifestList <|-- local_ManifestList + class ManifestList { + <> + +Add(repoName string) error + +Remove(repoName string) error + +Delete(additionalNames []string) error + +AnnotateManifest(manifestName string, opts AnnotateFields) error + +Save() error + } class remote_ManifestList { +NewManifestList(repoName string, keychain authn.Keychain) ManifestList @@ -179,16 +188,18 @@ classDiagram class local_ManifestList { +NewManifestList(repoName string, path string) ManifestList } - - class ManifestList { - <> - +Add(repoName string) error - +Remove(repoName string) error - +Save() error + + class AnnotateFields { + +String Architecture + +String OS + +String Variant } + + ManifestList <|-- remote_ManifestList + ManifestList <|-- local_ManifestList ``` -Two implementations: *remote* and *local* are propose, *remote* will take care of implementing the interaction with an OCI registry and *local* will deal with the local storage operations. +Two implementations: *remote* and *local* are proposed, *remote* will take care of implementing the interaction with an OCI registry and *local* will deal with the local storage operations. ### Component Diagram @@ -202,6 +213,15 @@ Using a [C4 component diagram](https://c4model.com/#ComponentDiagram), we can de ### Considerations +#### When a user wants to create a manifest list using a manifest outside the user's repo. + +Let's suppose the following user case: A user wants to create a manifest list `foo/my-manifest:my-tag` using a manifest outside his repository `foo` for example `other/external-manifest:latest`. + +In this case, pack will need to *copy* the external manifest `other/external-manifest:latest` into `foo` repository `foo/external-manifest:latest` and then uses this reference into the manifest list created. Pack should at least *warn* the user about this operation. + +#### When a user wants to create a manifest list referencing a manifest list + + # Migration [migration]: #migration From d5db1ec4abe1078ac074868cfcc7898faa44ed0f Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 6 Jul 2023 10:29:45 -0400 Subject: [PATCH 129/222] RFC 0121 [#235] Signed-off-by: Natalie Arellano --- ...ack-donation-to-cnb.md => 0121-kpack-donation-to-cnb.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-kpack-donation-to-cnb.md => 0121-kpack-donation-to-cnb.md} (99%) diff --git a/text/0000-kpack-donation-to-cnb.md b/text/0121-kpack-donation-to-cnb.md similarity index 99% rename from text/0000-kpack-donation-to-cnb.md rename to text/0121-kpack-donation-to-cnb.md index b708bc876..4316a97e6 100644 --- a/text/0000-kpack-donation-to-cnb.md +++ b/text/0121-kpack-donation-to-cnb.md @@ -3,10 +3,10 @@ - Name: kpack donation to CNB - Start Date: 2022-06-21 - Author(s): [Juan Bustamante](https://github.com/jjbustamante/) -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#235](https://github.com/buildpacks/rfcs/pull/235) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: N/A - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Summary From 73126685161ea96924b269db3640701183a4ee3f Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 11 Jul 2023 18:21:54 -0400 Subject: [PATCH 130/222] adding another consideration Signed-off-by: Juan Bustamante --- text/0000-pack-manifest-list-commands.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/text/0000-pack-manifest-list-commands.md b/text/0000-pack-manifest-list-commands.md index c1b709810..f91e215f8 100644 --- a/text/0000-pack-manifest-list-commands.md +++ b/text/0000-pack-manifest-list-commands.md @@ -215,12 +215,22 @@ Using a [C4 component diagram](https://c4model.com/#ComponentDiagram), we can de #### When a user wants to create a manifest list using a manifest outside the user's repo. -Let's suppose the following user case: A user wants to create a manifest list `foo/my-manifest:my-tag` using a manifest outside his repository `foo` for example `other/external-manifest:latest`. +As a pack user I want to create a manifest list `foo/my-manifest:my-tag` using a manifest outside my repository `foo` for example `other/external-manifest:latest`. -In this case, pack will need to *copy* the external manifest `other/external-manifest:latest` into `foo` repository `foo/external-manifest:latest` and then uses this reference into the manifest list created. Pack should at least *warn* the user about this operation. +The user invokes a command similar to: `pack manifest create foo/my-manifest:my-tag other/external-manifest:latest` + +In this case, pack will need to *copy* the external image `other/external-manifest:latest` into `foo` repository `foo/external-manifest:latest` and then uses this reference to create the manifest list. In such as case `pack` should: + - *warn* the user about this operation, for example with a message + - *ask the user for confirmation* before executing the operation #### When a user wants to create a manifest list referencing a manifest list +As a pack user I want to create a manifest list using a reference to another manifest list + +The user invokes a command similar to: `pack manifest create foo/my-manifest:my-tag foo/another-manifest:latest` + +In this case, pack should: + - add into the **manifests** array of objects a reference to the manifest index using the media-type `application/vnd.oci.image.index.v1+json` (nested index) # Migration [migration]: #migration From 399bdc35631f0b09251a9115919ad0a19722ecc2 Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Sat, 15 Jul 2023 11:17:35 +0100 Subject: [PATCH 131/222] Update text/0000-2023H2-roadmap.md Signed-off-by: Sambhav Kothari --- text/0000-2023H2-roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-2023H2-roadmap.md b/text/0000-2023H2-roadmap.md index a3f87e190..9a795cad6 100644 --- a/text/0000-2023H2-roadmap.md +++ b/text/0000-2023H2-roadmap.md @@ -95,7 +95,7 @@ Currently, Buildpack Authors have little to no tools around visibility with thei A platform operator can configure registry mirrors that lifecycle could use without needing for the manifest to have to point to it. This will allow a platform to reduce the risk of service operations from external registry sources and reduce public network bandwidth usage. The resulting image when taken off platform will also function without needing access to the registry mirror. #### kpack Donation -* Owner: @jjbustamante (Team Lead sponsor @hone) +* Owner: @jjbustamante (Team Lead sponsor @samj1912) * Links: [RFC](https://github.com/buildpacks/rfcs/pull/235) `kpack` is being proposed to be donated as an open source project in the Cloud Native Buildpacks' new Community Organization as a vendor neutral staging ground. This will give the project space to grow the project contributor base from multiple vendors. While work has started on this in H1, this item represents our commitment as a project to see this through and set this project up for success under the CNB governance umbrella. From 68de434b3a222eb78fd9eeafec52d9536d096ac6 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 17 Jul 2023 12:26:15 -0400 Subject: [PATCH 132/222] adding details to the commands Signed-off-by: Juan Bustamante --- text/0000-pack-manifest-list-commands.md | 153 ++++++++++++++++++++++- 1 file changed, 151 insertions(+), 2 deletions(-) diff --git a/text/0000-pack-manifest-list-commands.md b/text/0000-pack-manifest-list-commands.md index f91e215f8..65fc80059 100644 --- a/text/0000-pack-manifest-list-commands.md +++ b/text/0000-pack-manifest-list-commands.md @@ -57,8 +57,8 @@ The proposal is to add a new _experimental_ command `pack manifest` and differen - `pack manifest create` will create a local manifest list for annotating and pushing to a registry - `pack manifest annotate` will add additional information like os, arch or variant to an existing local manifest list - `pack manifest add` will add an image to an existing manifest list -- `pack manifest delete` will delete a manifest list from local storage -- `pack manifest rm` will remove an image from a manifest list +- `pack manifest remove` will delete a manifest list from local storage +- `pack manifest rm` will remove an image from a manifest list in the local storage - `pack manifest push` will push a manifest list to a registry - `pack manifest inspect` will show the manifest information stored in local storage @@ -232,6 +232,155 @@ The user invokes a command similar to: `pack manifest create foo/my-manifest:my- In this case, pack should: - add into the **manifests** array of objects a reference to the manifest index using the media-type `application/vnd.oci.image.index.v1+json` (nested index) +### Commands details + +#### Create a Manifest List + +Pack will create a manifest a local manifest, it should handle the following scenarios: +- IF user references a manifest list that already exists in a registry: In this case, pack will save a local copy of the remote manifest list, this is useful for updating (adding, updating or deleting) images +- IF user references a manifest list that doesn't exist in a registry: pack will create a local representation of the manifest list that will only save on the remote registry if the user publish it + +```bash +manifest create generates a manifest list for a multi-arch image + +Usage: + pack manifest create [ ... ] [flags] + +Examples: +pack manifest create cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe \ + cnbs/sample-package:hello-universe-windows + +Flags: + -f, --format string Format to save image index as ("OCI" or "V2S2") (default "v2s2") + --insecure Allow publishing to insecure registry + --publish Publish to registry + -r, --registry string Registry URL to publish the image index +``` + +#### Annotate (os/arch) a Manifest List + +Sometimes a manifest list could reference an image that doesn't specify the *architecture*, for example, [check](https://hub.docker.com/r/cnbs/sample-package/tags) our sample buildpack +packages. The `annotate` command allows users to update those values before pushing the manifest list a registry + +```bash +manifest annotate modifies a manifest list (Image index) and update the platform information for an image included in the manifest list. + +Usage: + pack manifest annotate [OPTIONS] [flags] + +Examples: +pack manifest annotate cnbs/sample-package:hello-universe-multiarch \ cnbs/sample-package:hello-universe --arch amd64 + +Flags: + --arch string Set the architecture + --os string Set the operating system + --variant string Set the architecture variant +``` + +#### Add an image to a Manifest List + +When a manifest list exits locally, user can add a new image to the manifest list using this command + +```bash +manifest add modifies a manifest list (Image index) and add a new image to the list of manifests. + +Usage: + pack manifest add [OPTIONS] [flags] + +Examples: +pack manifest add cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe-riscv-linux + +Flags: + --all add all of the contents to the local list (applies only if is an index) + --arch string Set the architecture + --os string Set the operating system + --variant string Set the architecture variant +``` + +#### Remove an image from a Manifest List + +In the opposite case, users can remove existing images from a manifest list + +```bash +Delete one or more manifest lists from local storage + +Usage: + pack manifest remove [manifest-list] [manifest-list...] [flags] + +Examples: +pack manifest delete cnbs/sample-package:hello-multiarch-universe + +Flags: + -h, --help Help for 'remove' +``` + +#### Remove a local Manifest List + +Sometimes users can just experiment with the feature locally and they want to discard all the local information created by pack. `rm` command just delete the *local* manifest list + +```bash +manifest remove will remove the specified image manifest if it is already referenced in the index + +Usage: + pack manifest rm [manifest-list] [manifest] [flags] + +Examples: +pack manifest rm cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe-windows + +Flags: + -h, --help Help for 'rm' + +``` + +#### Push a Manifest List to a remote registry + +Once a manifest list is ready to be publishe into the registry, the `push` command can be used + +```bash +manifest push pushes a manifest list (Image index) to a registry. + +Usage: + pack manifest push [OPTIONS] [flags] + +Examples: +pack manifest push cnbs/sample-package:hello-multiarch-universe + +Flags: + -f, --format string Manifest list type (oci or v2s2) to use when pushing the list (default is v2s2) + --insecure Allow publishing to insecure registry + -p, --purge Delete the manifest list or image index from local storage if pushing succeeds +``` + +#### Inspect a Manifest List + +Finally, the `inspect` command will help users to view how their local manifest list looks like + +```bash +manifest inspect shows the manifest information stored in local storage + +Usage: + pack manifest inspect [flags] + +Examples: +pack manifest inspect cnbs/sample-builder:multiarch + +Flags: + -h, --help Help for 'inspect' + +``` + +One important concern for users is to inspect the content of a multi-arch builder or buildpack if they are accessible behind a manifest list. The proposal is to implement a `platform` flag for commands: + +- `pack builder inspect` +- `pack buildpack inspect` + +The `--platform` flag specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). By default it will reference the host values. +The output of the commands should remain the same. + + # Migration [migration]: #migration From ef2ec1acf027db819a990713efff8738882512d4 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 19 Jul 2023 14:59:45 -0700 Subject: [PATCH 133/222] RFC 0122 --- .../{0000-2023H2-roadmap.md => 0122-2023H2-roadmap.md} | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) rename text/{0000-2023H2-roadmap.md => 0122-2023H2-roadmap.md} (96%) diff --git a/text/0000-2023H2-roadmap.md b/text/0122-2023H2-roadmap.md similarity index 96% rename from text/0000-2023H2-roadmap.md rename to text/0122-2023H2-roadmap.md index 9a795cad6..972ff9eef 100644 --- a/text/0000-2023H2-roadmap.md +++ b/text/0122-2023H2-roadmap.md @@ -3,11 +3,11 @@ - Name: 2023H2 Roadmap - Start Date: 2023-06-07 - Author(s): hone -- Status: Draft -- RFC Pull Request: (leave blank) -- CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) -- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) +- Status: Approved +- RFC Pull Request: [rfcs#286](https://github.com/buildpacks/rfcs/pull/286) +- CNB Pull Request: +- CNB Issue: N/A +- Supersedes: [RFC 0118](https://github.com/buildpacks/rfcs/blob/main/text/0118-2023H1-roadmap.md) # Summary [summary]: #summary From fc19b12708556d5c414169845eaec2ba642860a5 Mon Sep 17 00:00:00 2001 From: Domenico Luciani Date: Thu, 20 Jul 2023 15:45:13 +0200 Subject: [PATCH 134/222] RFC Flatten builders and buildpacks Signed-off-by: Domenico Luciani --- text/XXX-flatten-feature.md | 325 ++++++++++++++++++++++++++++++++++++ 1 file changed, 325 insertions(+) create mode 100644 text/XXX-flatten-feature.md diff --git a/text/XXX-flatten-feature.md b/text/XXX-flatten-feature.md new file mode 100644 index 000000000..779be67bd --- /dev/null +++ b/text/XXX-flatten-feature.md @@ -0,0 +1,325 @@ +# Meta +[meta]: #meta +- Name: Flatten builders and buildpacks +- Start Date: 2023-07-13 +- Author(s): @jjbustamante, @dlion +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +We propose to add new capabilities to the Pack tool that allow end users to reduce the number of Buildpack's layers in an OCI image by flattening according to their requirements. + +# Definitions +[definitions]: #definitions + +- Buildpack: A buildpack is a set of executables that inspects your app source code and creates a plan to build and run your application. +- Builder: A builder is an image that contains all the components necessary to execute a build. A builder image is created by taking a build image and adding a lifecycle, buildpacks, and files that configure aspects of the build including the buildpack detection order and the location(s) of the run image + +# Motivation +[motivation]: #motivation + +- Why should we do this? + +There is a limit in the number of layer an image can have, at least on Docker, which is *127*, this feature has being request by the community, issue [#1595](https://github.com/buildpacks/pack/issues/1595), as a workaround to solve error thrown by docker when the limit is reached + +- What use cases does it support? + +Buildpacks provider like Paketo have composite buildpacks with several layers, when they pull many of those together into a builder, hitting the layer limit for a container image happens very often. A feature for the Buildpack author to group the buildpacks by any attribute will allow them to squash those groups into one layer and reduce their total number of layers, avoiding the layer limit. + +- What is the expected outcome? + +When Buildpacks Authors execute commands like: + +`pack builder create ... ` or +`pack buildpack package ... ` + +The final OCI image artifact (A) SHOULD contain layers blobs with more than *one* buildpack according to the configuration provided by the user. If we compare an artifact (B) created *without* `` then: + +$numberOfBuildpackLayers(A) \leq numberOfBuildpackLayers(B)$ + +A and B MUST be otherwise interchangeable, only differing by their number of layers. + + +# What it is +[what-it-is]: #what-it-is + +The proposal is to include new experimental flags to the following commands on pack: + +- `pack builder create` +- `pack buildpack package` + +The new flags will move from experimental status to supported status when maintainers deem it appropriate. + +The new flags to be included are: + +- `--flatten` will flatten all the Buildpacks in one layer +- `--flatten --depth=` If there are composite buildpack to be flatten, `depth` is the high in the buildpack dependency tree to start flattening the buildpacks +- `--flatten --flatten-exclude=` will flatten all the Buildpacks in one layer excepts the ones specified under the `flatten-exclude` flag +- `--flatten --flatten-include=` will flatten the Buildpacks specified after the `flatten-include` flag into a single layer. Can be used more than once, with each use resulting in a single layer. + +We also need to define how a platform implementor needs to consume a flatten buildpackpage or builder. + +- When a platform consumes an OCI image artifact, they will need to inspect each buildpack layer blob and determine if the blob contains more than one buildpack, in such as case, they will need to process those buildpacks correctly. + + +# How it Works +[how-it-works]: #how-it-works + +Let's say we have a composite buildpack (CB1) with the following dependency tree: +```mermaid +flowchart TD + A[CB1] + A --> B[G1] + A --> C[G2] + B --> BPA[BP1] + B --> BPFOO[BP2] + B --> BPC[BP4] + C --> BPD[BP1] + C --> BPBAR[BP3] + C --> BPE[BP4] +``` + +Until now, when a buildpack like this is being shipped into an OCI image every individual buildpack is being saved in one layer, as a result we will have: + +$$ + layer_1 = [CB_1] \\ + layer_2 = [G_1] \\ + layer_3 = [BP_1] \\ + layer_4 = [BP_2] \\ + layer_5 = [BP_4] \\ + layer_6 = [G_2] \\ + layer_7 = [BP_3] \\ + total = \text{7 layers} +$$ + +Noticed that duplicated buildpacks are cleaned up. + +We can use the new `flatten` flag to reduce the number of OCI image layers used by the buildpacks in different ways. + +* `--flatten`: +Will flatten all buildpacks into one layer, the result will be: + +```mermaid +classDiagram + class Layer1 { + CB1 + G1 + BP1 + BP2 + BP4 + G2 + BP3 + } +``` + + +$$ + total = \text{1 layer} +$$ + +--- +* `--flatten --flatten-exclude=` i.e. `--flatten-exclude=` +Will flatten all buildpacks in 1 layer except for`BP2`, the result will be + +```mermaid +classDiagram + class Layer1 { + CB1 + G1 + BP1 + BP4 + G2 + BP3 + } + class Layer2 { + BP2 + } +``` + +$$ + total = \text{2 layers} +$$ + +--- + +* `--flatten --depth=0` +Will flatten all layers at the root (0) level, which is similar to flatten all, the result will be: + +```mermaid +classDiagram + class Layer1 { + CB1 + G1 + BP1 + BP2 + BP4 + G2 + BP3 + } +``` + +$$ + total = \text{1 layer} +$$ + +--- +Increasing the depth number will allow to flatten the tree deeper, +for example setting `--depth=1`, the result will be: + +```mermaid +classDiagram + class Layer1 { + CB1 + } + class Layer2 { + G1 + BP1 + BP2 + BP4 + } + class Layer3 { + G2 + BP3 + } +``` + + +$$ + total = \text{3 layers} +$$ + + +--- + +* `--flatten --flatten-include=` i.e. `--flatten --flatten-include= --flatten-include=` + +Will group the given buildpacks into one layer and keep the other ones as single layers buildpacks, the result will be: + +```mermaid +classDiagram + class Layer1 { + CB1 + } + class Layer2 { + G1 + } + class Layer3 { + BP1 + BP2 + } + class Layer4 { + G2 + } + class Layer5 { + BP3 + BP4 + } +``` + + +$$ + total = \text{5 layers} +$$ + + +# Migration +[migration]: #migration + +The current [distribution spec](https://github.com/buildpacks/spec/blob/main/distribution.md#buildpackage) defines: + +``` +Each buildpack layer blob MUST contain a single buildpack at the following file path: + +/cnb/buildpacks/// +``` + +A Buildpackage flattened with this new feature would not be consumable by older platform implementations because they are not expecting to find more than one buildpack on a blob layer. + + + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +Distributing a buildpackage with more than one buildpack in a layer blob adds complexity to platform implementors because they will need to do the opposite process when consuming the OCI images. + +It will add complexity to Buildpack Authors processes because they will need to think how to group their buildpacks together in a layer blob in an efficient way avoiding for example network issues when pulling new blobs from a registry. Size, frequency of change and other variables must be taken into consideration when buildpacks are flatten. + +It will create artifacts that are not consumable by older platforms. + + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + +Some other alternatives mentioned are: squashing by the buildpack size or squashing a CNB Builder when the number of layers is reaching the limit, but those ideas, do not deal with the main problems of distributing more than one buildpack in a layer blob. + + +- Why is this proposal the best? + +Not sure if it is the best, but way to solve the `layer limit error` is to optimize the uses of the layer in a OCI image. + +- What is the impact of not doing this? + +Buildpack Authors and Platform Operators will keep seeing the layer limit error. + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? + +We think the `--depth` command MAY not be helpful, after implementing a PoC. Should we get rid of this flag? + + + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +Maybe, but it's not clear about this. + + + +# History +[history]: #history + +% \ No newline at end of file From c33e34be8340f4715c8dc1c3c8f4970eb5f585a2 Mon Sep 17 00:00:00 2001 From: Domenico Luciani Date: Tue, 1 Aug 2023 15:41:47 +0200 Subject: [PATCH 135/222] Renamed rfc Signed-off-by: Domenico Luciani --- text/{XXX-flatten-feature.md => 0000-flatten-feature.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{XXX-flatten-feature.md => 0000-flatten-feature.md} (100%) diff --git a/text/XXX-flatten-feature.md b/text/0000-flatten-feature.md similarity index 100% rename from text/XXX-flatten-feature.md rename to text/0000-flatten-feature.md From 61e166c3b7111acffb87f9ad8684a0f0f79ff14c Mon Sep 17 00:00:00 2001 From: Domenico Luciani Date: Tue, 1 Aug 2023 17:11:53 +0200 Subject: [PATCH 136/222] Updated the RFC based on the feedback we got from our stakelholders Signed-off-by: Domenico Luciani --- text/0000-flatten-feature.md | 143 +++++------------------------------ 1 file changed, 20 insertions(+), 123 deletions(-) diff --git a/text/0000-flatten-feature.md b/text/0000-flatten-feature.md index 779be67bd..19924580d 100644 --- a/text/0000-flatten-feature.md +++ b/text/0000-flatten-feature.md @@ -38,7 +38,7 @@ When Buildpacks Authors execute commands like: `pack builder create ... ` or `pack buildpack package ... ` -The final OCI image artifact (A) SHOULD contain layers blobs with more than *one* buildpack according to the configuration provided by the user. If we compare an artifact (B) created *without* `` then: +The final OCI image artifact (A) SHOULD contain layers blobs with more than *one* buildpack according to the configuration provided by the user. If we compare an artifact (B) created *without* `` then: $numberOfBuildpackLayers(A) \leq numberOfBuildpackLayers(B)$ @@ -57,10 +57,7 @@ The new flags will move from experimental status to supported status when mainta The new flags to be included are: -- `--flatten` will flatten all the Buildpacks in one layer -- `--flatten --depth=` If there are composite buildpack to be flatten, `depth` is the high in the buildpack dependency tree to start flattening the buildpacks -- `--flatten --flatten-exclude=` will flatten all the Buildpacks in one layer excepts the ones specified under the `flatten-exclude` flag -- `--flatten --flatten-include=` will flatten the Buildpacks specified after the `flatten-include` flag into a single layer. Can be used more than once, with each use resulting in a single layer. +- `--flatten=` will flatten the Buildpacks specified after the `flatten` flag into a single layer. Can be used more than once, with each use resulting in a single layer. We also need to define how a platform implementor needs to consume a flatten buildpackpage or builder. @@ -86,119 +83,23 @@ flowchart TD Until now, when a buildpack like this is being shipped into an OCI image every individual buildpack is being saved in one layer, as a result we will have: -$$ - layer_1 = [CB_1] \\ - layer_2 = [G_1] \\ - layer_3 = [BP_1] \\ - layer_4 = [BP_2] \\ - layer_5 = [BP_4] \\ - layer_6 = [G_2] \\ - layer_7 = [BP_3] \\ - total = \text{7 layers} $$ - -Noticed that duplicated buildpacks are cleaned up. - -We can use the new `flatten` flag to reduce the number of OCI image layers used by the buildpacks in different ways. - -* `--flatten`: -Will flatten all buildpacks into one layer, the result will be: - -```mermaid -classDiagram - class Layer1 { - CB1 - G1 - BP1 - BP2 - BP4 - G2 - BP3 - } -``` - - -$$ - total = \text{1 layer} -$$ - ---- -* `--flatten --flatten-exclude=` i.e. `--flatten-exclude=` -Will flatten all buildpacks in 1 layer except for`BP2`, the result will be - -```mermaid -classDiagram - class Layer1 { - CB1 - G1 - BP1 - BP4 - G2 - BP3 - } - class Layer2 { - BP2 - } -``` - -$$ - total = \text{2 layers} -$$ - ---- - -* `--flatten --depth=0` -Will flatten all layers at the root (0) level, which is similar to flatten all, the result will be: - -```mermaid -classDiagram - class Layer1 { - CB1 - G1 - BP1 - BP2 - BP4 - G2 - BP3 - } -``` - -$$ - total = \text{1 layer} +layer_1 = [CB_1] \\ +layer_2 = [G_1] \\ +layer_3 = [BP_1] \\ +layer_4 = [BP_2] \\ +layer_5 = [BP_4] \\ +layer_6 = [G_2] \\ +layer_7 = [BP_3] \\ +total = \text{7 layers} $$ ---- -Increasing the depth number will allow to flatten the tree deeper, -for example setting `--depth=1`, the result will be: +Noticed that duplicated buildpacks are cleaned up. -```mermaid -classDiagram - class Layer1 { - CB1 - } - class Layer2 { - G1 - BP1 - BP2 - BP4 - } - class Layer3 { - G2 - BP3 - } -``` - - -$$ - total = \text{3 layers} -$$ - - ---- - -* `--flatten --flatten-include=` i.e. `--flatten --flatten-include= --flatten-include=` +We can use the new `flatten` flag to reduce the number of OCI image layers used by the buildpacks in different ways. -Will group the given buildpacks into one layer and keep the other ones as single layers buildpacks, the result will be: +* `--flatten=` i.e. `--flatten= --flatten=`: + Will group the given buildpacks into one layer and keep the other ones as single layers buildpacks, the result will be: ```mermaid classDiagram @@ -222,9 +123,11 @@ classDiagram ``` -$$ - total = \text{5 layers} $$ +total = \text{5 layers} +$$ + +--- # Migration @@ -270,20 +173,14 @@ Not sure if it is the best, but way to solve the `layer limit error` is to optim - What is the impact of not doing this? -Buildpack Authors and Platform Operators will keep seeing the layer limit error. +Buildpack Authors and Platform Operators will keep seeing the layer limit error. # Prior Art [prior-art]: #prior-art Discuss prior art, both the good and bad. -# Unresolved Questions -[unresolved-questions]: #unresolved-questions - -- What parts of the design do you expect to be resolved before this gets merged? - -We think the `--depth` command MAY not be helpful, after implementing a PoC. Should we get rid of this flag? - +--- \ No newline at end of file From 0728af7829c09c632dc3fd8329bdca8029614b03 Mon Sep 17 00:00:00 2001 From: Domenico Luciani Date: Mon, 4 Sep 2023 10:34:40 +0200 Subject: [PATCH 139/222] Cleaned the rfc removing anything not related to a Builder Signed-off-by: Domenico Luciani --- text/0000-flatten-feature.md | 55 +++++++++++++++++------------------- 1 file changed, 26 insertions(+), 29 deletions(-) diff --git a/text/0000-flatten-feature.md b/text/0000-flatten-feature.md index 3e5994235..bf3339d3c 100644 --- a/text/0000-flatten-feature.md +++ b/text/0000-flatten-feature.md @@ -1,6 +1,6 @@ # Meta [meta]: #meta -- Name: Flatten builders and buildpacks +- Name: Flatten builders - Start Date: 2023-07-13 - Author(s): @jjbustamante, @dlion - Status: Draft @@ -12,7 +12,9 @@ # Summary [summary]: #summary -We propose to add new capabilities to the Pack tool that allow end users to reduce the number of Buildpack's layers in an OCI image by flattening according to their requirements. +We propose to add new capabilities to the Pack tool that allow end-users to reduce the number of Buildpack's layers in a Builder by flattening some Buildpacks according to their requirements. + +This RFC mainly focus on applying this strategy to Builders only. # Definitions [definitions]: #definitions @@ -21,7 +23,7 @@ We propose to add new capabilities to the Pack tool that allow end users to redu - Builder: A builder is an image that contains all the components necessary to execute a build. A builder image is created by taking a build image and adding a lifecycle, buildpacks, and files that configure aspects of the build including the buildpack detection order and the location(s) of the run image - Component Buildpack: A component buildpack is a buildpack containing `/bin/detect` and `/bin/build` executables. Component buildpacks implement the [Buildpack Interface](https://github.com/buildpacks/spec/blob/main/buildpack.md). - Composite Buildpack: A composite buildpack is a buildpack containing an order definition in `buildpack.toml`. Composite buildpacks do not contain `/bin/detect` or `/bin/build` executables. They MUST be [resolvable](https://github.com/buildpacks/spec/blob/main/buildpack.md#order-resolution) into a collection of component buildpacks. -- Buildpackage: A buildpackage is a [distributable](https://github.com/buildpacks/spec/blob/main/distribution.md) artifact that contains a buildpack. +- Buildpackage: A buildpackage is a [distributable](https://github.com/buildpacks/spec/blob/main/distribution.md) artifact that contains a buildpack. # Motivation [motivation]: #motivation @@ -32,16 +34,15 @@ There is a limit in the number of layer an image can have, at least on Docker, w - What use cases does it support? -Buildpacks provider like Paketo have composite buildpacks with several layers, when they pull many of those together into a builder, hitting the layer limit for a container image happens very often. A feature for the Buildpack author to group the buildpacks by any attribute will allow them to squash those groups into one layer and reduce their total number of layers, avoiding the layer limit. +Buildpacks provider like Paketo have Composite Buildpacks with several layers, when they pull many of those together into a Builder, hitting the layer limit for a container image happens very often. A feature for the Builder author to group the Buildpacks by any attribute will allow them to squash those groups into one layer and reduce their total number of layers, avoiding the layer limit. - What is the expected outcome? -When Buildpacks Authors execute commands like: +When Builder Authors execute the command: -`pack builder create ... ` or -`pack buildpack package ... ` +`pack builder create ... ` -The final OCI image artifact (A) SHOULD contain layers blobs with more than *one* buildpack according to the configuration provided by the user. If we compare an artifact (B) created *without* `` then: +The final Builder (A) SHOULD contain layers blobs with more than *one* buildpack according to the configuration provided by the user. If we compare an artifact (B) created *without* `` then: $numberOfBuildpackLayers(A) \leq numberOfBuildpackLayers(B)$ @@ -51,26 +52,25 @@ A and B MUST be otherwise interchangeable, only differing by their number of lay # What it is [what-it-is]: #what-it-is -The proposal is to include new experimental flags to the following commands on pack: +The proposal is to include a new experimental flag to the following command on Pack: - `pack builder create` -- `pack buildpack package` -The new flags will move from experimental status to supported status when maintainers deem it appropriate. +The new flag will move from experimental status to supported status when maintainers deem it appropriate. -The new flags to be included are: +The new flag to be included is: - `--flatten=` will flatten the Buildpacks specified after the `flatten` flag into a single layer. Can be used more than once, with each use resulting in a single layer. -We also need to define how a platform implementor needs to consume a flatten buildpackpage or builder. +We also need to define how a Platform implementor needs to consume a flattened Builder. -- When a platform consumes an OCI image artifact, they will need to inspect each buildpack layer blob and determine if the blob contains more than one buildpack, in such as case, they will need to process those buildpacks correctly. +- When a Platform consumes a Builder, they will need to inspect each Buildpack layer blob and determine if the blob contains more than one Buildpack, in such as case, they will need to process those Buildpacks correctly. # How it Works [how-it-works]: #how-it-works -Let's say we have a composite buildpack (CB1) with the following dependency tree: +Let's say we have a Composite Buildpack (CB1) with the following dependency tree: ```mermaid flowchart TD A[CB1] @@ -84,7 +84,7 @@ flowchart TD C --> BPE[BP4] ``` -Until now, when a buildpack like this is being shipped into an OCI image every individual buildpack is being saved in one layer, as a result we will have: +Until now, when a Buildpack like this is being shipped into a Builder every individual Buildpack is being saved in one layer, as a result we will have: $$ layer_1 = [CB_1] \\ @@ -97,12 +97,12 @@ layer_7 = [BP_3] \\ total = \text{7 layers} $$ -Noticed that duplicated buildpacks are cleaned up. +Noticed that duplicated Buildpacks are cleaned up. -We can use the new `flatten` flag to reduce the number of OCI image layers used by the buildpacks in different ways. +We can use the new `flatten` flag to reduce the number of Builder layers used by the buildpacks in different ways. * `--flatten=` i.e. `--flatten= --flatten=`: - Will group the given buildpacks into one layer and keep the other ones as single layers buildpacks, the result will be: + Will group the given Buildpacks into one layer and keep the other ones as single layers Buildpacks, the result will be: ```mermaid classDiagram @@ -136,6 +136,7 @@ $$ # Migration [migration]: #migration + The current [distribution spec](https://github.com/buildpacks/spec/blob/main/distribution.md#buildpackage) defines: ``` @@ -144,7 +145,7 @@ Each buildpack layer blob MUST contain a single buildpack at the following file /cnb/buildpacks/// ``` -A Buildpackage and a Builder flattened with this new feature would not be consumable by older platform implementations because they are not expecting to find more than one buildpack on a blob layer. +A Builder flattened with this new feature would not be consumable by older platform implementations because they are not expecting to find more than one buildpack on a blob layer. +- Status: Approved - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) @@ -110,7 +110,7 @@ classDiagram CB1 } class Layer2 { - G1 + G1 } class Layer3 { BP1 @@ -182,7 +182,7 @@ Discuss prior art, both the good and bad. --- -% \ No newline at end of file +--->% diff --git a/text/0124-pack-manifest-list-commands.md b/text/0124-pack-manifest-list-commands.md index 65fc80059..0e2a7f5a0 100644 --- a/text/0124-pack-manifest-list-commands.md +++ b/text/0124-pack-manifest-list-commands.md @@ -3,7 +3,7 @@ - Name: Manifest List Commands for Pack - Start Date: 2023-04-19 - Author(s): [Juan Bustamante](https://github.com/jjbustamante) -- Status: Draft +- Status: Approved - RFC Pull Request: (leave blank) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) @@ -22,7 +22,7 @@ This RFC proposes to create a new set of CRUD commands in pack to handle manifes [definitions]: #definitions - Image Manifest: The image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. See [spec](https://github.com/opencontainers/image-spec/blob/main/manifest.md) -- Image Index: The image index is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. See [spec](https://github.com/opencontainers/image-spec/blob/main/image-index.md) +- Image Index: The image index is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. See [spec](https://github.com/opencontainers/image-spec/blob/main/image-index.md) # Motivation [motivation]: #motivation @@ -38,10 +38,10 @@ Or the conversations around this topic in our [slack channel](https://cloud-nati - What use cases does it support? -Currently, buildpack authors can build and package their buildpacks for different OS and Architectures, but when they distribute them the URI for a buildpack can’t disambiguate, they need to use different tags to differentiate between them. This makes harder for users to consume those Buildpacks. +Currently, buildpack authors can build and package their buildpacks for different OS and Architectures, but when they distribute them the URI for a buildpack can’t disambiguate, they need to use different tags to differentiate between them. This makes harder for users to consume those Buildpacks. The solution is to share a single URI for all the different OS and Architectures, and the way to do that is using a manifest list. -Adding commands to support the operations to handle the manifest list will allow buildpack authors to migrate their existing buildpacks to support multi-arch, without afecting their current existing process. +Adding commands to support the operations to handle the manifest list will allow buildpack authors to migrate their existing buildpacks to support multi-arch, without afecting their current existing process. - What is the expected outcome? @@ -62,17 +62,17 @@ The proposal is to add a new _experimental_ command `pack manifest` and differen - `pack manifest push` will push a manifest list to a registry - `pack manifest inspect` will show the manifest information stored in local storage -Our target user affected by the feature is: **Buildpack Authors**. Let's see some examples of how this feature will work. +Our target user affected by the feature is: **Buildpack Authors**. Let's see some examples of how this feature will work. -Currently, if we check [sample-packages](https://hub.docker.com/r/cnbs/sample-package/tags) at dockerhub we will notice that we have a composed buildpack called `hello-universe` and we offer two tags to support different architectures: -- `cnbs/sample-package:hello-universe` for linux and -- `cnbs/sample-package:hello-universe-windows`. +Currently, if we check [sample-packages](https://hub.docker.com/r/cnbs/sample-package/tags) at dockerhub we will notice that we have a composed buildpack called `hello-universe` and we offer two tags to support different architectures: +- `cnbs/sample-package:hello-universe` for linux and +- `cnbs/sample-package:hello-universe-windows`. Let's suppose our linux version is called `cnbs/sample-package:hello-universe-linux` to keep the same naming convention, but we will keep it as it is for simplicity. If we want to distribute the `hello-universe` buildpack for any **architecture/os/variant** combination we need to use a tool outside the CNB ecosystem to create a manifest list. With the proposed experimental commands on pack we can do: ```bash -$ pack manifest create cnbs/sample-package:hello-multiarch-universe \ - cnbs/sample-package:hello-universe \ +$ pack manifest create cnbs/sample-package:hello-multiarch-universe \ + cnbs/sample-package:hello-universe \ cnbs/sample-package:hello-universe-windows ``` @@ -100,11 +100,11 @@ By default, the command will create a manifest list in the local storage using t "os": "windows", "architecture": "" } - } + } ] } ``` -The idea to save the manifest list locally is to allow the user to update the manifest before pushing it to a registry, +The idea to save the manifest list locally is to allow the user to update the manifest before pushing it to a registry, in this case, we need to define the **architecture** field because it is empty in our examples. We can use the `pack manifest annotate` command to add the architecture information: @@ -167,7 +167,7 @@ The proposal is to implement an abstraction of an OCI *Image Index* and expose i ## Image Index Abstraction -A new high level abstraction to represent an OCI Image Index is proposed, similar to the [Image](https://github.com/buildpacks/imgutil/blob/main/image.go) interface exposed in *imgutil* repository, +A new high level abstraction to represent an OCI Image Index is proposed, similar to the [Image](https://github.com/buildpacks/imgutil/blob/main/image.go) interface exposed in *imgutil* repository, we proposed a new *ManifestList* interface to expose the behavior of an OCI Image Index. ```mermaid @@ -182,21 +182,21 @@ classDiagram } class remote_ManifestList { - +NewManifestList(repoName string, keychain authn.Keychain) ManifestList + +NewManifestList(repoName string, keychain authn.Keychain) ManifestList } class local_ManifestList { +NewManifestList(repoName string, path string) ManifestList } - + class AnnotateFields { +String Architecture +String OS +String Variant } - + ManifestList <|-- remote_ManifestList - ManifestList <|-- local_ManifestList + ManifestList <|-- local_ManifestList ``` Two implementations: *remote* and *local* are proposed, *remote* will take care of implementing the interaction with an OCI registry and *local* will deal with the local storage operations. @@ -213,7 +213,7 @@ Using a [C4 component diagram](https://c4model.com/#ComponentDiagram), we can de ### Considerations -#### When a user wants to create a manifest list using a manifest outside the user's repo. +#### When a user wants to create a manifest list using a manifest outside the user's repo. As a pack user I want to create a manifest list `foo/my-manifest:my-tag` using a manifest outside my repository `foo` for example `other/external-manifest:latest`. @@ -258,7 +258,7 @@ Flags: -r, --registry string Registry URL to publish the image index ``` -#### Annotate (os/arch) a Manifest List +#### Annotate (os/arch) a Manifest List Sometimes a manifest list could reference an image that doesn't specify the *architecture*, for example, [check](https://hub.docker.com/r/cnbs/sample-package/tags) our sample buildpack packages. The `annotate` command allows users to update those values before pushing the manifest list a registry @@ -377,7 +377,7 @@ One important concern for users is to inspect the content of a multi-arch builde - `pack builder inspect` - `pack buildpack inspect` -The `--platform` flag specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). By default it will reference the host values. +The `--platform` flag specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). By default it will reference the host values. The output of the commands should remain the same. @@ -412,7 +412,7 @@ The impact of not doing this is that users will need to use other tools to creat [prior-art]: #prior-art These features are inspired in similar commands in other tools like: -- [docker](https://docs.docker.com/engine/reference/commandline/manifest/) +- [docker](https://docs.docker.com/engine/reference/commandline/manifest/) - [podman](https://docs.podman.io/en/v3.2.0/markdown/podman-manifest-create.1.html) # Unresolved Questions From a68ed1dae7b127aa59458c61d050423588be5d10 Mon Sep 17 00:00:00 2001 From: Kritka Sahni <122665407+kritkasahni-google@users.noreply.github.com> Date: Tue, 17 Oct 2023 16:03:56 -0700 Subject: [PATCH 146/222] Create 0000-creator-skip-sbom.md Signed-off-by: Kritka Sahni <122665407+kritkasahni-google@users.noreply.github.com> --- text/0000-creator-skip-sbom.md | 131 +++++++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 text/0000-creator-skip-sbom.md diff --git a/text/0000-creator-skip-sbom.md b/text/0000-creator-skip-sbom.md new file mode 100644 index 000000000..6e01fcf56 --- /dev/null +++ b/text/0000-creator-skip-sbom.md @@ -0,0 +1,131 @@ +# Meta +[meta]: #meta +- Name: Enable CNB_SKIP_SBOM IN /cnb/lifecycle/creator +- Start Date: (fill in today's date: 2023-10-17) +- Author(s): kritkasahni-google +- Status: Draft +- RFC Pull Request: +- CNB Pull Request: +- CNB Issue: +- Supersedes: N/A + +# Summary +[summary]: #summary + +Enable CNB_SKIP_SBOM IN /cnb/lifecycle/creator to skip restoring SBOM layer from previous app image. We support CNB_SKIP_LAYERS in analyzer which does the same thing and we should support the same in creator also. + +# Definitions +[definitions]: #definitions +* lifecycle: software that orchestrates a CNB build. +* creator: executes all the lifecycle phases one by one in order. +* analyzer: lifecycle phase that restores SBOM layer from previous app image. +* restorer: lifecycle phase that restores layers from cache. +* SBOM: a software bill of materials (SBOM) is a list of all the components that make up the app image. + +# Motivation +[motivation]: #motivation + +To skip restoring SBOM layer from previous image when platform executes lifecycle by calling /cnb/lifecycle/creator. Restoring SBOM layer from previous app image can cause degraded build latency but if buildpack logic does not rely on SBOM from previous app image then should be able to skip restoring it. + +# What it is +[what-it-is]: #what-it-is + +CNB_SKIP_LAYERS is used by /cnb/lifecycle/analyzer to skip restoring SBOM layer from previous app image. +Need a similar mechanism for /cnb/lifecyle/creator specifically to skip restoring only the SBOM layer. + +The target personas affected by this change are: + + - **buildpack user**: if buildpacks don't rely on reusing SBOM layer then buildpack user should ideally see improved build latency by skipping SBOM restoration but reusing other layers from previous app image. + - **Platform implementors**: they will choose to skip restoring SBOM by providing CNB_SKIP_SBOM to trigger /cnb/lifecycle/creator. + + +# How it Works +[how-it-works]: #how-it-works + +Similar to how CNB_SKIP_LAYERS is handled in analyzer whether SBOM needs to be [restored](https://github.com/buildpacks/lifecycle/blob/292aa492a72f4e180bb92d109a73ebf7c8a0451d/phase/analyzer.go#L38) or [not](https://github.com/buildpacks/lifecycle/blob/292aa492a72f4e180bb92d109a73ebf7c8a0451d/phase/analyzer.go#L30) today, CNB_SKIP_SBOM will be be handled in same way in analyzer. +At the platform level, it would be input same way as CNB_SKIP_LAYERS [here](https://github.com/buildpacks/lifecycle/blob/292aa492a72f4e180bb92d109a73ebf7c8a0451d/platform/defaults.go#L184) and [handled](https://github.com/buildpacks/lifecycle/blob/main/platform/lifecycle_inputs.go#L82) like:- + + +``` + var skipSBOM bool + if boolEnv(EnvSkipSBOM){ + skipSBOM = true + } +``` + +In the analyzer, + +``` +analyzer := &Analyzer{ + Logger: logger, + SBOMRestorer: &layer.NopSBOMRestorer{}, + PlatformAPI: f.platformAPI, + } + + ... + if f.platformAPI.AtLeast("0.8") && !inputs.SkipLayers && !inputs.SkipSBOM { + analyzer.SBOMRestorer = &layer.DefaultSBOMRestorer{ + LayersDir: inputs.LayersDir, + Logger: logger, + } + } +``` + +# Migration +[migration]: #migration + +CNB_SKIP_SBOM/ will be an optional input to /cnb/lifecycle/creator, and will be false by default. We maybe need to add a new API option for buildpack users, to choose whether this should be enabled. + +# Drawbacks +[drawbacks]: #drawbacks + +N/A + +# Alternatives +[alternatives]: #alternatives + +Platforms that execute lifecycle today via /cnb/lifecycle/creator are unable to skip restoring SBOM layer from previous app image unless they skip reusing previous app image entirely. + +# Prior Art +[prior-art]: #prior-art + +We already support enabling CNB_SKIP_LAYERS in /cnb/lifecycle/analyzer and /cnb/lifecycle/restorer, and CNB_SKIP_RESTORE in /cnb/lifecycle/creator. +* CNB_SKIP_LAYERS in /cnb/lifecycle/analyzer to skip restoring SBOM from previous app image. +* CNB_SKIP_LAYERS in /cnb/lifecycle/restorer to skip reusing previous app image layers entirely. +* CNB_SKIP_RESTORE in /cnb/lifecycle/creator to skips restoring SBOM plus all other layers entirely from previous app image. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes +This new feature will affect the API of [Create](https://buildpacks.io/docs/concepts/components/lifecycle/create/) phase by adding the following fields. + +Back to API changes, we will add a new flag to control this. + +| Input | Environment Variable | DefaultValue | Description | +|----------------|-----------------------|--------------|----------------------------------------------| +| `` | `CNB_SKIP_SBOM` | `false` | Skip SBOM restoration | + +# History +[history]: #history + + From 9752e5939fd1a9111cba903cdd8e8cf2a0d2c8a9 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Mon, 23 Oct 2023 10:15:57 -0400 Subject: [PATCH 147/222] RFC 0125 [#291] Signed-off-by: Natalie Arellano --- ...parallel-export.md => 0125-lifecycle-parallel-export.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-lifecycle-parallel-export.md => 0125-lifecycle-parallel-export.md} (98%) diff --git a/text/0000-lifecycle-parallel-export.md b/text/0125-lifecycle-parallel-export.md similarity index 98% rename from text/0000-lifecycle-parallel-export.md rename to text/0125-lifecycle-parallel-export.md index bc5cbca3a..0d193a8ab 100644 --- a/text/0000-lifecycle-parallel-export.md +++ b/text/0125-lifecycle-parallel-export.md @@ -3,10 +3,10 @@ - Name: Export App Image and Cache Image in Parallel - Start Date: 2023-08-26 - Author(s): ESWZY -- Status: Draft -- RFC Pull Request: +- Status: Approved +- RFC Pull Request: [rfcs#291](https://github.com/buildpacks/rfcs/pull/291) - CNB Pull Request: [lifecycle#1167](https://github.com/buildpacks/lifecycle/pull/1167) -- CNB Issue: +- CNB Issue: N/A - Supersedes: N/A # Summary From 91a8eaac4204a89c9301fe836785a17f80ba3336 Mon Sep 17 00:00:00 2001 From: Pavel Busko Date: Mon, 9 Oct 2023 14:41:32 +0200 Subject: [PATCH 148/222] Add RFC for extension layers Co-authored-by: Ralf Pannemans Signed-off-by: Ralf Pannemans Co-authored-by: Pavel Busko Signed-off-by: Pavel Busko --- text/0000-extension-layer.md | 90 ++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 text/0000-extension-layer.md diff --git a/text/0000-extension-layer.md b/text/0000-extension-layer.md new file mode 100644 index 000000000..e337fb0b0 --- /dev/null +++ b/text/0000-extension-layer.md @@ -0,0 +1,90 @@ +# Meta +[meta]: #meta +- Name: Add extension layer to exchange data +- Start Date: 2023-10-09 +- Author(s): [c0d1ngm0nk3y](https://github.com/c0d1ngm0nk3y), [pbusko](https://github.com/pbusko) +- Status: Draft +- RFC Pull Request: +- CNB Pull Request: +- CNB Issue: +- Related: [RFC#0105 Support Dockerfiles](https://github.com/buildpacks/rfcs/blob/main/text/0105-dockerfiles.md) +- Supersedes: N/A + +# Summary +[summary]: #summary + +This RFC introduces support for Extension Layers to allow data transfer between the build environment and the Kaniko execution. + +# Motivation +[motivation]: #motivation + +This change allows extensions to possess their own layers to utilize during the generation/extend process. Additionally, it ensures that extension output does not inadvertently interfere with other extension or buildpack layers during the build, and it does not unintentionally become part of the final application image. + +This would allow distroless run images to be extended. + +# What it is +[what-it-is]: #what-it-is + +This follows up on RFC-0105 and proposes that during the execution of the extension's `bin/generate`, an extension is allowed to write arbitrary data to its exclusive layer. This data then becomes accessible during the execution of the `extend` phase via Kaniko context. The content of these extension-specific layers is ignored at build and launch time, it serves only the extension phase. + +# How it Works +[how-it-works]: #how-it-works + +Before execution of the `bin/generate`, the lifecycle will create a distinct writable layer for each extension which passed detection. The extensions can then write to these layers, and the Kaniko context is set to the corresponding layer during the `extend` phase instead of the `` directory. The Extension Layers will not be included in the final image by the lifecycle. + +The location of the Extension Layer will be provided to the `bin/generate` via additional environment variable `CNB_EXT_LAYER_DIR`. + +# Migration +[migration]: #migration + +Since we would change the Kaniko context, this would be breaking for existing extensions. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +The workspace would no longer be the Kaniko context. But the benefit of having it in the first place seems quite limited anyway. + +# Alternatives +[alternatives]: #alternatives + +- Allow multi-stage Dockerfiles + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- Should the `bin/generate` be executed during the `extend` phase instead of the `detect` phase? +- The Kaniko context might consist of two folders: `` and the Extension Layer for better compatibility. + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +This RFC requires changes to the layers metadata and the `extend` phase: + +- layer metadata needs ti ubducate if a layer is a "extension layer" +- env variable with the layer location for the extension to write to +- kaniko context should be the extension layer and not the workspace + + \ No newline at end of file From 54fd7b04d74ab771cc5f158d4532beeb1add3063 Mon Sep 17 00:00:00 2001 From: Pavel Busko Date: Wed, 25 Oct 2023 14:24:19 +0200 Subject: [PATCH 149/222] add examples Signed-off-by: Pavel Busko --- text/0000-extension-layer.md | 48 ++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/text/0000-extension-layer.md b/text/0000-extension-layer.md index e337fb0b0..6fc860499 100644 --- a/text/0000-extension-layer.md +++ b/text/0000-extension-layer.md @@ -25,26 +25,56 @@ This would allow distroless run images to be extended. # What it is [what-it-is]: #what-it-is -This follows up on RFC-0105 and proposes that during the execution of the extension's `bin/generate`, an extension is allowed to write arbitrary data to its exclusive layer. This data then becomes accessible during the execution of the `extend` phase via Kaniko context. The content of these extension-specific layers is ignored at build and launch time, it serves only the extension phase. +This follows up on RFC-0105 and proposes that during the execution of the extension's `./bin/generate`, an extension is allowed to write arbitrary data to the `context` folder within its exclusive layer. This data then becomes accessible during the execution of the `extend` phase via Kaniko build context. The content of these extension-specific layers is ignored at build and launch time, it serves only the extension phase. # How it Works [how-it-works]: #how-it-works -Before execution of the `bin/generate`, the lifecycle will create a distinct writable layer for each extension which passed detection. The extensions can then write to these layers, and the Kaniko context is set to the corresponding layer during the `extend` phase instead of the `` directory. The Extension Layers will not be included in the final image by the lifecycle. +- New root directory `/layers-ext` is introduced which contains extension layers. +- Before execution of the `./bin/generate`, the lifecycle will create a distinct writable layer `/layers-ext/` for each extension which passed detection. +- The `/layers-ext/` is provided to the `./bin/generate` as `` directory. +- In addition to the files specified in [RFC#0105](https://github.com/buildpacks/rfcs/blob/main/text/0105-dockerfiles.md), the extension may create the `/context` folder with an arbitrary content. +- If the folder `/context` is present it will be set as Kaniko build context during the `extend` phase instead of the `` directory. +- If the folder `/context` is not present, Kaniko build context defaults to the `` folder. + +The `/layers-ext` will not be included in the final image by the lifecycle. -The location of the Extension Layer will be provided to the `bin/generate` via additional environment variable `CNB_EXT_LAYER_DIR`. +### Example: Extend distroless run image with Debian packages. + +This example extension would allow to install `tar` package on the run image without package manager (distroless image). The extension contains `./bin/generate` and `./bin/custom-installer` file, which installs `.deb` files. + +##### `./bin/generate` + +```bash +#!/bin/sh + +mkdir -p ${CNB_OUTPUT_DIR}/context + +cp ${CNB_EXTENSION_DIR}/bin/custom-installer ${CNB_OUTPUT_DIR}/context/ +curl -o ${CNB_OUTPUT_DIR}/context/tar.deb http://security.ubuntu.com/ubuntu/pool/main/t/tar/tar_1.34+dfsg-1ubuntu0.1.22.04.1_amd64.deb + +cat >> "${CNB_OUTPUT_DIR}/run.Dockerfile" <` and the Extension Layer for better compatibility. +- Should the `./bin/generate` be executed during the `extend` phase instead of the `detect` phase? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes This RFC requires changes to the layers metadata and the `extend` phase: -- layer metadata needs ti ubducate if a layer is a "extension layer" -- env variable with the layer location for the extension to write to -- kaniko context should be the extension layer and not the workspace +- allow optional folder `/context` with an arbitrary content to be provided by extension. +- if the `/context` is present, kaniko context should be set to this folder instead of the ``. +- Status: Approved - RFC Pull Request: - CNB Pull Request: - CNB Issue: From 88c024f529c56828e74c520e53b51d83e8e5211c Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 14 Sep 2023 17:04:31 -0500 Subject: [PATCH 156/222] Initial version Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 944 ++++++++++++++++++++ 1 file changed, 944 insertions(+) create mode 100644 text/0000-multiarch-builders-and-package.md diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md new file mode 100644 index 000000000..aeaa88d5f --- /dev/null +++ b/text/0000-multiarch-builders-and-package.md @@ -0,0 +1,944 @@ +# Meta +[meta]: #meta +- Name: Multi-arch support for builders and buildpack packages +- Start Date: 2023-09-14 +- Author(s): @jjbustamante +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +The problem for adding support for multi-arch buildpacks can be divided into three parts: +1. Support buildpack authors to **migrate their existing buildpacks** to support multi-arch. +2. Support buildpack authors to **create new buildpacks and builders** that handle multi-arch from the beginning. +3. Support application developers to **create application images** using multi-arch buildpacks and builders. + +The purpose of this RFC is to solve the statement 2, adding the capability to the commands: + +- `pack buildpack package` +- `pack builder create` + +to create individuals OCI images artifacts per each os and arch (builders and buildpack packages) and handle the creation for the [image index,](https://github.com/opencontainers/image-spec/blob/master/image-index.md) +that combines them into one single consumable tag for end-users. + +# Definitions +[definitions]: #definitions + +- Buildpack: A buildpack is a set of executables that inspects your app source code and creates a plan to build and run your application. +- Builder: A builder is an image that contains all the components necessary to execute a build. A builder image is created by taking a build image and adding a lifecycle, buildpacks, and files that configure aspects of the build including the buildpack detection order and the location(s) of the run image +- Image Manifest: The image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. See [spec](https://github.com/opencontainers/image-spec/blob/main/manifest.md) +- Image Index: The image index is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. See [spec](https://github.com/opencontainers/image-spec/blob/main/image-index.md) + +# Motivation +[motivation]: #motivation + +- Why should we do this? + +The uses of ARM architecture in the cloud and edge computing has been growing rapidly. The CNCF community has been also growing in the last years, and there is a need to support multi-arch for all the projects. The buildpacks community is not an exception, issues like: +- [It would be nice to support easily creating a manifest list packages and builders](https://github.com/buildpacks/pack/issues/1460) +- [Provide a way to specify desired platform when creating packages and builders](https://github.com/buildpacks/pack/issues/1459) +- [Multi arch image build support](https://github.com/buildpacks/pack/issues/1570) + +Or the conversations around this topic in our [slack channel](https://cloud-native.slack.com/archives/C032LNSMY0P), even the [talk at Kubecon NA 2022](https://www.youtube.com/watch?v=Sdr5axlOnDI&list=PLj6h78yzYM2O5aNpRM71NQyx3WUe1xpTn&index=76) demonstrate the interest from the community in this feature. + +- What use cases does it support? + +Currently, buildpack authors can build and package their buildpacks for different OS and Architectures, but when they distribute them the URI for a buildpack can’t disambiguate, +they need to use different tags to differentiate between them. Tools like `docker buildx imagetools create` helps to create an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) to combine them but increasing their build pipelines complexity. +For example, take a look at this recent blog [post](https://deploy-preview-53--elegant-borg-5bd068.netlify.app/blog/steps-we-took-for-a-basic-arm64-support-in-buildpacks) or the [instructions](https://github.com/dmikusa/paketo-arm64/) created from @dmikusa to build an ARM64 builder. + +For those buildpack authors that are using `cross-compile` languages like [go](https://go.dev/) or [rust](https://www.rust-lang.org/) or maybe bash scripts, adding the capability to `pack buildpack package` and `pack builder create` to create multi-arch images +and also handle the creation of an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will simplify their CI/CD pipelines and make the experience more suitable. + +- What is the expected outcome? + +`pack buildpack package` and `pack builder create` commands will be updated in a way that they will handle the creation of multi-arch OCI images + +# What it is +[what-it-is]: #what-it-is + +The end-users for this proposal are **Buildpack authors**, we expect to improve their user experience when creating multi-architecture buildpacks and builders as follows + +## Multi-arch example + +Let's suppose a **Buildpack author** has a `buildpack.toml` created with a content similar to this one: + +```toml +# Buildpack API version +api = "0.12" + +# Buildpack ID and metadata +[buildpack] + id = "examples/my-multiarch-buildpack" + version = "0.0.1" + +# List of targets operating systems, architectures and versions + +[[targets]] +os = "linux" +arch = "amd64" + +[[targets]] +os = "linux" +arch = "arm64" + +[[targets]] +os = "windows" +arch = "amd64" + +[[targets.distributions]] +name = "windows" +versions = ["10.0.20348.1970"] +``` + +And organizes the binaries according to their os/arch with a structure similar to this one: + +```bash +my-multiarch-buildpack +├── buildpack.toml +├── linux-amd64 +│ └── bin +│ ├── build +│ └── detect +├── linux-arm64 +│ └── bin +│ ├── build +│ └── detect +└── windows-amd64 + └── 10.0.20348.1970 + └── bin + ├── build.bat + └── detect.bat +``` + +Now `pack` will be able to package them separately for each os/arch family, following our [guide](https://buildpacks.io/docs/buildpack-author-guide/package-a-buildpack/) +we will need to create a `package.toml` file. + +The `package.toml` file will be similar to the one describe in our documentation: + +```toml +[buildpack] +uri = "examples/my-multiarch-buildpack" +# OR a .tgz with the previous folder structure +uri = "my-multiarch-buildpack.tgz" +``` + +But in this case, we actually **remove** the [platform](https://buildpacks.io/docs/reference/config/package-config/) section because it will be taken from the `buildpack.toml`. + +Packaging a multi-arch buildpack will require the output to be **publish** to a registry or **saved on disk** in OCI layout format. + +```bash +pack buildpack package my-buildpack --config ./package.toml --publish --multi-arch +# Or +pack buildpack package my-buildpack.cnb --config ./package.toml --format file --multi-arch +``` + +In these cases each `target` entry corresponds to a different buildpack image that is exported into an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) + +### Package a multi-arch Builder + +In case of packing a **Builder**, we assume the following premises: + +1. Buildpack authors migrated their `builder.toml` and [remove the stack](https://github.com/buildpacks/pack/issues/1303) concept. +2. Multi-architecture `build`, `run` images and `buildpacks` are available for baking into the **Builder**. + +A sample `builder.toml` file looks like: + +```toml +# Buildpacks to include in builder, these buildpacks MUST be multi-arch and point to Image Index +[[buildpacks]] +uri = "" + +[run] +# Runtime images - in case of multi-arch images it must point to Image Index +[[run.images]] +image = "index.docker.io/paketobuildpacks/run-jammy-tiny:latest" + +[build] +# This image is used at build-time, in case of multi-arch images it must point to Image Index +image = "docker.io/paketobuildpacks/build-jammy-tiny:0.2.3" + +[[targets]] +os = "linux" +arch = "amd64" + +[[targets]] +os = "linux" +arch = "arm64" +``` + +As we can see, the proposal is based on the assumption that the `run-image`, `build-image` and `buildpacks` to include +in the builder are **multi-arch artifacts**, and we can reach them by reading an +[image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) + +```bash +pack builder create my-jammy-builder --config ./builder.toml \ + --multi-arch \ + --publish +``` + +In this case, because `targets` are specified and `--multi-arch` is being used `pack` will follow the builder creation process for **each provided platform**, +pulling the correct (based on os/arch) buildpacks, build and run images and creating different builders images that are +exported and combined into an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) + +# How it Works +[how-it-works]: #how-it-works + +## Buildpack Package + +As a quick summary, our current process to create a buildpack package involves: + +- The end-users defined `os` for the OCI image using the [package.toml](https://buildpacks.io/docs/reference/config/package-config/) when they want to export a buildpack package. +- The only values allowed are `linux` and `windows` and by default when is not present, `linux` is being used. +- When exporting to daemon, the `docker.OSType` must be equal to `platform.os` + +### To keep compatibility + +We propose: +- Deprecate the `platform.os` field from [package.toml](https://buildpacks.io/docs/reference/config/package-config/). It will be removed after two pack releases with the new feature +- When `platform.os` is present in [package.toml](https://buildpacks.io/docs/reference/config/package-config/), throw a warning messages indicating the field will be removed +and `--platform` flag must be used +- When `platform.os` is not present [package.toml](https://buildpacks.io/docs/reference/config/package-config/) and `--platform` flag is not used, throw a warning messages indicating +a new `--platform` flag is available +- Keep doing our current process to package a buildpack + +### To improve user experience + +We propose: +- Add a new `--platform` flag with format `[os][/arch][/variant]:[name@version]` to build for a particular target, once the `platform.os` field is removed, +this will be the way for end-users to specify the platform for which they want to create single OCI artifact. +> **Note** +> We want to avoid duplicated information, we expect to have `targets` with all the platform supported by a buildpack + +- Add a new boolean `--multi-arch` flag to indicate pack it must create multiples OCI artifacts and combine them with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md), +this flag must be used in conjunction with `--publish` or `--format file` flags and error out when `daemon` is selected. +> **Note** +> I got feedback to avoid creating an Image Index if users are not expecting that, the flag is an acknowledgment for creating the index + +- A new folder structure to organize the buildpacks binaries for multi-arch images +```bash +. +├── buildpack.toml +└── {os}-{arch} // optional + └── {variant} // optional + ├── {version-1} // optional + │ └── bin + │ ├── build + │ └── detect + └── {version-2} // optional + └── bin + ├── build + └── detect +``` +> **Note** +> For cross-compile buildpacks like Paketo, it looks easy to add a step to their Makefile to compile and separate the binaries following this structure. + +Based on the [RFC-0096](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) we replaced the concept of `stacks` in `buildpack.toml` by a `target` which looks like: + +```toml +[[targets]] +os = "" +arch = "" +variant = "" +[[targets.distributions]] +name = "" +versions = [""] +``` +- When `--multi-arch` is enabled + - When `--publish` or `--format file` is specified + - `pack` will read `targets` from `buildpack.toml` + - For each `target` an OCI image will be created, following our current process + - `pack` will try to infer a folder structure similar to the one show above, and if it is found, those are the binaries to include in the OCI image + - If more than 1 OCI image was created, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them + +### Examples + +Let's use some examples to explain the expected behavior in different use cases + +#### Buildpacks authors do not use targets + +##### `platform.os` is not present at `package.toml` + +This seems to be the case for [Paketo Buildpacks](https://github.com/paketo-buildpacks/java/blob/main/package.toml) +or [Heroku](https://github.com/heroku/buildpacks-jvm/blob/main/meta-buildpacks/java/package.toml). A simplified version of their folder structures is: + +```bash +├── bin +│ ├── build +│ └── detect +├── buildpack.toml +└── package.toml +``` + +In these cases, the expected output will be similar to: + +```bash +pack buildpack package --config ./package.toml --publish +Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Successfully published package and saved to registry + +# Or +pack buildpack package --config ./package.toml --format file +Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Successfully created package and saved to file +``` + +We will keep `linux` as the default platform, and we don't expect end-users to change anything else on their side. + +In case, Buildpack authors tries to use the new folder structure like: + +```bash +├── buildpack.toml +├── linux-amd64 +│ └── bin +│ ├── build +│ └── detect +├── linux-arm64 +│ └── bin +│ ├── build +│ └── detect +└── package.toml +``` + +The current `pack` version creates an OCI artifact copying all the files on it. + +![](https://hackmd.io/_uploads/BkGkZ_mla.png) + +But I think this buildpack can't be executed because `./bin/build` or `./bin/detect` can't be found. We could add some validations and throw an error, +but I do not see any value on doing that. + +> **Important** +> New folder structure is not useful if Buildpack Authors don't remove stacks and migrate to use targets +> + +Trying to use the new flags + +```bash +pack buildpack package --config ./package.toml --publish --multi-arch +Error: 'targets' or 'platforms' must be defined when creating a multi-architecture buildpack + +# Or +pack buildpack package --config ./package.toml --format file --multi-arch +Error: 'targets' or 'platforms' must be defined when creating a multi-architecture buildpack +``` + +In these cases, we don't have enough information to create a multi-arch buildpack, and pack must fail its execution. + +A valid way to re-write the command could be: + +```bash +pack buildpack package --config ./package.toml --publish --platform linux/arm64 --platform linux/amd64 --multi-arch +A multi-arch buildpack package will be created for platforms: 'linux/amd64', 'linux/arm64' +Successfully published package and saved to registry + +# Or +pack buildpack package --config ./package.toml --format file --platform linux/arm64 --platform linux/amd64 --multi-arch +A multi-arch buildpack package will be created for platforms: 'linux/amd64', 'linux/arm64' +Successfully created package and saved to file +``` + +In these cases, two OCI images, with the same binaries, will be created and pushed into the registry, for each image the configuration file will be +created with the correct `os` and `architecture` and an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them + +what about creating a single image for a new platform? + +```bash +pack buildpack package --config ./package.toml --publish --platform linux/arm64 +Successfully published package and saved to registry + +# Or +pack buildpack package --config ./package.toml --format file --platform linux/arm64 +Successfully created package and saved to file +``` + +In these cases, pack will create buildpack package image with the provided binaries and a [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) +with the following platform: + +```json +{ + "architecture": "arm64", + "os": "linux" +} +``` + +##### `platform.os` is present at `package.toml` + +These cases are similar to the previous one, but we can change the warning message. + +```bash +pack buildpack package --config ./package.toml --publish +Warning: 'platform.os' field in package.toml will be deprecated, use new '--platform' flag to set target platform +Successfully published package and saved to registry + +# Or +pack buildpack package --config ./package.toml --format file +Warning: 'platform.os' field in package.toml will be deprecated, use new '--platform' flag to set target platform +Successfully created package and saved to file +``` + +#### Buildpacks authors use targets + +We can divide the problem in two main scenarios: Buildpack authors use or not use the new folder structure. + +Let's start with the first one, which is the natural path for Buildpack Authors to migrate from `stacks` to `targets`. Let's suppose a buildpack folder structure like: + +```bash +├── bin +│ ├── build +│ └── detect +├── buildpack.toml +└── package.toml +``` + +And a `buildpack.toml` with `targets` defined as: + +```toml +[[targets]] +os = "linux" +arch = "amd64" + +[[targets]] +os = "linux" +arch = "arm64" +``` + +How could Buildpack Authors set which `os/arch` to use?, currently `pack` doesn't care about `targets` and +it sets `linux` as the `os` in the final OCI image. We can keep this behavior, and show warning messages to end-users, +this will avoid breaking the current behavior. + +```bash +pack buildpack package --config ./package.toml --publish +Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' +Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Successfully published package and saved to registry + +# Or +pack buildpack package --config ./package.toml --format file +Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' +Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Successfully created package and saved to file + +# Or +pack buildpack package --config ./package.toml +Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' +Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Successfully created package and saved to docker daemon +``` + +The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: + +```json +{ + "architecture": "", + "os": "linux" +} +``` + +On the other hand, when end-users use the new `platform` flag + +```bash +pack buildpack package --config ./package.toml --publish --platform linux/arm64 +Successfully published package and saved to registry +``` + +The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: + +```json +{ + "architecture": "amd64", + "os": "linux" +} +``` + +Trying to create a buildpack package for an unsupported platform should throw an error + +```bash +pack buildpack package --config ./package.toml --publish --platform windows/arm64 +Error: platform 'windows/arm64' is not supported in buildpack targets, check your 'buildpack.toml' +``` + +Let's go one more step further, and check what will happen if Buildpack Authors use the new `--multi-arch` flag + +```bash +pack buildpack package --config ./package.toml --publish --multi-arch +Info: A multi-arch buildpack package will be created for platforms: 'linux/amd64', 'linux/arm64' +Successfully published package and saved to registry +``` + +In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be +created with the correct `os` and `architecture` and +an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them, +with a content similar to: + +```json + +{ + "manifests": [ + { + "digest": "sha256:b492494d8e0113c4ad3fe4528a4b5ff89faa5331f7d52c5c138196f69ce176a6", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": { + "architecture": "amd64", + "os": "linux" + }, + "size": 424 + }, + { + "digest": "sha256:2589fe6bcf90466564741ae0d8309d1323f33b6ec8a5d401a62d0b256bcc3c37", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": { + "architecture": "arm", + "os": "linux" + }, + "size": 424 + } + ], + "mediaType": "application/vnd.oci.image.index.v1+json", + "schemaVersion": 2 +} +``` +> **Important** +> In this case, the layer blob for the OCI image will have the same bin/build and bin/detect binaries. This could be helpful +> for bash buildpacks for example. + +There is a downside for this case, how can we do it if we have windows and linux scripts? +One way to do it could be just copying the .bat scripts along with the linux scripts, something like: + +```bash +├── bin +│ ├── build +│ ├── build.bat +│ ├── detect +│ └── detect.bat +├── buildpack.toml +└── package.toml +``` + +But we will end up distributing windows and linux scripts together. + +Finally, let's check some examples for the second scenario, when Buildpack Authors migrated from `stacks` to `targets` and +they want to take advantage of the new multi-architecture capabilities, let's use our original folder structure: + +```bash +├── buildpack.toml +├── linux-amd64 +│ └── bin +│ ├── build +│ └── detect +├── linux-arm64 +│ └── bin +│ ├── build +│ └── detect +└── windows-amd64 + └── 10.0.20348.1970 + └── bin + ├── build.bat + └── detect.bat +``` + +And a `buildpack.toml` with the following `targets` defined: + +```toml +[[targets]] +os = "linux" +arch = "amd64" + +[[targets]] +os = "linux" +arch = "arm64" + +[[targets]] +os = "windows" +arch = "amd64" + +[[targets.distributions]] +name = "windows" +versions = ["10.0.20348.1970"] +``` + +If the Buildpack Author wants to create a single buildpack package they will use the `platform` flag + +```bash +pack buildpack package --config ./package.toml --publish --platform linux/arm64 +Successfully published package and saved to registry +``` + +In this case, `pack` will determine the folder `linux-arm64` exists, and that must be the root folder for the buildpack. +`pack` will copy `buildpack.toml` from the root and include it, something like: + +```bash +linux-arm64 +├── bin +│ ├── build +│ └── detect +└── buildpack.toml // Must be copy on the fly +``` + +Similar to previous cases, the OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: + +```json +{ + "architecture": "arm64", + "os": "linux" +} +``` + +A fully multi-arch buildpack could be created as follows: + +```bash +pack buildpack package --config ./package.toml --publish --multi-arch +Info: A multi-arch buildpack package will be created for platforms: 'linux/amd64', 'linux/arm64', 'windows/amd64' +Successfully published package and saved to registry +``` + +In this case, three OCI images will be created and pushed into the registry, for each image the configuration file will be +created with the correct platform: `os` and `architecture`, +an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them, +with a content similar to: + +```json + +{ + "manifests": [ + { + "digest": "sha256:b492494d8e0113c4ad3fe4528a4b5ff89faa5331f7d52c5c138196f69ce176a6", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": { + "architecture": "amd64", + "os": "linux" + }, + "size": 424 + }, + { + "digest": "sha256:2589fe6bcf90466564741ae0d8309d1323f33b6ec8a5d401a62d0b256bcc3c37", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": { + "architecture": "arm", + "os": "linux" + }, + "size": 424 + }, + { + "digest": "sha256:ed1a67bb47f3c35d782293229127ac1f8d64873a131186c49fe079dada0fa7e0", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": { + "architecture": "amd64", + "os": "windows", + "os.version": "10.0.20348.1970" + }, + "size": 424 + } + ], + "mediaType": "application/vnd.oci.image.index.v1+json", + "schemaVersion": 2 +} +``` + +> **Important** +> In this case, each image has different binaries according to the platform and the new folder structure + +## Builder + +Similar to how we did it for the `buildpack package`, lets summaries, our current process to create a **Builder**: + +- We read the `builder.toml` and fetch the `run.image`, currently we didn't specify the `platform`, **host** `os` is being used. +- We read the `builder.toml` and fetch the `build.image`, currently we didn't specify the `platform`, **host** `os` is being used. +- We create a **base builder** from the `build.image`. +- We read the `os` and `architecture` from the **base builder**, fetch a `lifecycle` image that matches the platform (Note: lifecycle is already publish behind an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md)) +- We add `Buildpacks` and `Extensions` to the **base builder**, because we don't define `architecture` for them, we fetch matching the **base builder** `os` +- More logic and finally the **Builder** image is created + +### To keep compatibility + +We propose: + +- When `stacks` is present in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/), throw a warning message indicating the field is deprecated and +it will be removed +- When `targets` is not present in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/), throw a warning messages indicating + a new `--platform` flag is available +- Keep doing our current process to create a builder + +### To improve user experience + +We propose: + +- Add `targets` section to the `builder.toml` schema, this will keep consistency for end-users to understand how to define multi-architecture. The new schema will be +similar to: +```toml +# Buildpacks to include in builder, +# MUST point to an Image Index that matches targets +[[buildpacks]] +uri = "" + +[run] +# Runtime images +# MUST point to an Image Index that matches targets +[[run.images]] +image = "" + +[build] +# This image is used at build-time +# MUST point to an Image Index that matches targets +image = "" + +# Platforms to support with the Builder +[[targets]] +os = "" +arch = "" +variant = "" +[[targets.distributions]] +name = "" +versions = [""] +``` +- Add a new `--platform` optional flag with format `[os][/arch][/variant]:[name@version]` to create a builder for a particular target, this will help end-users to specify the platform for which they want to create single OCI artifact. +- Add a new boolean `--multi-arch` flag to indicate pack it must create multiples OCI artifacts and combine them with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md), + this flag must be used in conjunction with `--publish` or `--format file` flags and error out when `daemon` is selected. + +### Examples + +Let's use some examples to explain the expected behavior in different use cases + +#### `Targets` are not present in `builder.toml` + +This is probably the case for most of the Buildpack Authors, for example [Paketo](https://github.com/paketo-buildpacks/builder-jammy-tiny/blob/main/builder.toml), lets suppose a +`buildpack.toml` like: + +```toml +# Buildpacks to include in builder +[[buildpacks]] +uri = "" + +# Order used for detection +[[order]] +[[order.group]] +id = "" +version = "" + +[stack] +id = "io.buildpacks.samples.stacks.jammy" +build-image = "cnbs/sample-base-build:jammy" +run-image = "cnbs/sample-base-run:jammy" +``` +Or we include `build` and `run` images + +```toml +# Base images used to create the builder +[build] +image = "cnbs/sample-base-build:jammy" +[run] +[[run.images]] +image = "cnbs/sample-base-run:jammy" +``` + +In these cases, the expected output will be similar to: + +```bash +pack builder create --config ./builder.toml +Warning: "stack" has been deprecated, prefer "targets" instead: https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md +Warning: A new '--platform' flag is available to set the target platform for the builder, using 'linux/amd64' as default +Successfully created builder image +Tip: Run pack build --builder to use this builder +``` +We expect the command to keep working as today, the builder image will be created but some **warning** messages will be +printed to help end-users to check for new updates, maybe link to a migration guide? + +Trying to use the new flags: + +```bash +pack builder create --config ./builder.toml --platform linux/arm64 +Warning: "stack" has been deprecated, prefer "targets" instead: https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md +Warning: creating a builder for platform "linux/arm64" but "targets" is not defined, update your "builder.toml" to include "targets" +Successfully created builder image +Tip: Run pack build --builder to use this builder +``` + +Pulling operations will be configured to use `linux/arm64` as platform, +the OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: + +```json +{ + "architecture": "arm64", + "os": "linux" +} +``` + +What about multi-architecture builders? + +```bash +pack builder create --config ./builder.toml --multi-arch --publish +Error: 'targets' or 'platforms' must be defined when creating a multi-architecture builder +``` + +In this case, pack doesn't have enough information to create a multi-arch builder and fail its execution. + +Using `platform` flag: + +```bash +pack builder create --config ./builder.toml \ + --multi-arch \ + --plaform linux/amd64 \ + --platform linux/arm64 \ + --publish +Successfully created builder image +Tip: Run pack build --builder to use this builder +``` + +In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be +created with the correct platform: `os` and `architecture`, +an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them + +#### `Targets` are present in `builder.toml` + +Let's suppose a `builder.toml` similar to this one: + +```toml +[[buildpacks]] +id = "samples/java-maven" +version = "0.0.1" +uri = "" + +[[order]] +[[order.group]] +id = "samples/java-maven" +version = "0.0.1" + +[build] +image = "cnbs/sample-base-build:jammy" +[run] +[[run.images]] +image = "cnbs/sample-base-run:jammy" + +[[targets]] +os = "linux" +arch = "amd64" + +[[targets]] +os = "linux" +arch = "arm64" +``` + +Let's suppose we execute the command in a host "linux/amd64" machine + +```bash +pack builder create --config ./builder.toml +Info: creating a builder for platform "linux/amd64" +Successfully created builder image +Tip: Run pack build --builder to use this builder +``` + +`platform` flag is not defined, we keep our current behavior and detect the host `os` and `architecture`. Because a +`target` matches the host `platform` it is used to create the builder. + +Trying to use the new flags: + +```bash +pack builder create --config ./builder.toml --platform linux/arm64 +Info: creating a builder for platform "linux/arm64" +Successfully created builder image +Tip: Run pack build --builder to use this builder +``` + +Pulling operations will be configured to use `linux/arm64` as platform, +the OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: + +```json +{ + "architecture": "arm64", + "os": "linux" +} +``` + +What about multi-architecture builders? + +```bash +pack builder create --config ./builder.toml --multi-arch --publish +Info: A multi-arch builder will be created for platforms: 'linux/amd64', 'linux/arm64' +Successfully created builder image +Tip: Run pack build --builder to use this builder +``` + + +Using `platform` flag: + +```bash +pack builder create --config ./builder.toml \ + --multi-arch \ + --plaform linux/amd64 \ + --platform linux/arm64 \ + --publish +Info: A multi-arch builder will be created for platforms: 'linux/amd64', 'linux/arm64' +Successfully created builder image +Tip: Run pack build --builder to use this builder +``` + +In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be +created with the correct platform: `os` and `architecture`, +an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them + + +# Migration +[migration]: #migration + +1. Align with the Stack removal [plan](https://docs.google.com/document/d/1bExpr31U5R5yQ6fncpl5YcdosWVwYcXgkt12vE-lpvU/edit) +2. Deprecate the `platform.os` field from [package.toml](https://buildpacks.io/docs/reference/config/package-config/) + - We don't want to break current behavior, but we do want community migrate to the new approach +3. Update docs to explain the new functionality, blog posts or any other useful media communicate the message +4. Remove `platform.os` support on `pack` + +# Drawbacks +[drawbacks]: #drawbacks + +- New complexity will be added into `pack` + +# Alternatives +[alternatives]: #alternatives + +- Do nothing, Buildpack Authors can keep using other tools like`docker buildx imagetools create` or `crane` to update the `architecture` in their Manifest files or +create [image indexes](https://github.com/opencontainers/image-spec/blob/master/image-index.md). +- Do not deprecate `platform.os` field from [package.toml](https://buildpacks.io/docs/reference/config/package-config/) and add more fields to get the same result, instead of flags + - I didn't explore this idea + +# Prior Art +[prior-art]: #prior-art + +- Stack Removal [RFC #096](https://github.com/buildpacks/rfcs/blob/jjbustamante/feature/multi-arch-phase-2/text/0096-remove-stacks-mixins.md) +- This RFC is a continuation of the work started with the proposal to add commands to handle manifest list in pack, see the [RFC](https://github.com/buildpacks/rfcs/pull/283) +- Paketo [RFC #288](https://github.com/paketo-buildpacks/rfcs/pull/288) to publish multi-arch buildpacks + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- If I already have an [image indexe](https://github.com/opencontainers/image-spec/blob/master/image-index.md) created and I want to add support for new plaform, how do I do it? +- What are the intermediate images for each platform named/called? +- What happen if I want to exclude some buildpack for a particular target? +- What happen if I want to include the same file or folder for every image, do I have to copy then inside the {os}-{arch} folder? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +NA + +# History +[history]: #history + + From af7811b6b0dc3e0b42be181b9f922b8c2e1ef706 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Tue, 7 Nov 2023 15:58:00 -0500 Subject: [PATCH 157/222] Change flag --platform to use --target, and adding a field to override the path for binaries in buildpack.toml Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 236 +++++++++++--------- 1 file changed, 135 insertions(+), 101 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index aeaa88d5f..eb5348395 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -65,7 +65,7 @@ The end-users for this proposal are **Buildpack authors**, we expect to improve ## Multi-arch example -Let's suppose a **Buildpack author** has a `buildpack.toml` created with a content similar to this one: +Let's suppose a **Buildpack author** has a `buildpack.toml` updated to include `targets` as follows: ```toml # Buildpack API version @@ -93,26 +93,33 @@ arch = "amd64" [[targets.distributions]] name = "windows" versions = ["10.0.20348.1970"] + +# Stacks (deprecated) the buildpack will work with +[[stacks]] +id = "*" ``` And organizes the binaries according to their os/arch with a structure similar to this one: ```bash my-multiarch-buildpack +. ├── buildpack.toml -├── linux-amd64 -│ └── bin -│ ├── build -│ └── detect -├── linux-arm64 -│ └── bin -│ ├── build -│ └── detect -└── windows-amd64 - └── 10.0.20348.1970 - └── bin - ├── build.bat - └── detect.bat +├── linux +│ ├── amd64 +│ │ └── bin +│ │ ├── build +│ │ └── detect +│ └── arm64 +│ └── bin +│ ├── build +│ └── detect +└── windows + └── amd64 + └── windows@10.0.20348.1970 + └── bin + ├── build.bat + └── detect.bat ``` Now `pack` will be able to package them separately for each os/arch family, following our [guide](https://buildpacks.io/docs/buildpack-author-guide/package-a-buildpack/) @@ -132,18 +139,21 @@ But in this case, we actually **remove** the [platform](https://buildpacks.io/do Packaging a multi-arch buildpack will require the output to be **publish** to a registry or **saved on disk** in OCI layout format. ```bash -pack buildpack package my-buildpack --config ./package.toml --publish --multi-arch +pack buildpack package my-buildpack --config ./package.toml --publish # Or -pack buildpack package my-buildpack.cnb --config ./package.toml --format file --multi-arch +pack buildpack package my-buildpack.cnb --config ./package.toml --format file ``` +> **Important** +> pack will determine a multi-arch buildpack package is being created because there are more than one target defined. + In these cases each `target` entry corresponds to a different buildpack image that is exported into an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) ### Package a multi-arch Builder In case of packing a **Builder**, we assume the following premises: -1. Buildpack authors migrated their `builder.toml` and [remove the stack](https://github.com/buildpacks/pack/issues/1303) concept. +1. Buildpack authors updated their `builder.toml` to include the new `targets` fields defined in this RFC. 2. Multi-architecture `build`, `run` images and `buildpacks` are available for baking into the **Builder**. A sample `builder.toml` file looks like: @@ -169,6 +179,14 @@ arch = "amd64" [[targets]] os = "linux" arch = "arm64" + +# Stack (deprecated) +[stack] +id = "io.buildpacks.stacks.jammy.tiny" +# This image is used at build-time +build-image = "docker.io/paketobuildpacks/build-jammy-tiny:0.2.3" +# This image is used at runtime +run-image = "index.docker.io/paketobuildpacks/run-jammy-tiny:latest" ``` As we can see, the proposal is based on the assumption that the `run-image`, `build-image` and `buildpacks` to include @@ -176,12 +194,12 @@ in the builder are **multi-arch artifacts**, and we can reach them by reading an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) ```bash -pack builder create my-jammy-builder --config ./builder.toml \ - --multi-arch \ - --publish +pack builder create my-jammy-builder --config ./builder.toml --publish ``` +> **Important** +> Similar to the `buildpack package`, pack will determine a multi-arch builder must be created based on the multiple targets defined. -In this case, because `targets` are specified and `--multi-arch` is being used `pack` will follow the builder creation process for **each provided platform**, +In this case `pack` will follow the builder creation process for **each provided target**, pulling the correct (based on os/arch) buildpacks, build and run images and creating different builders images that are exported and combined into an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) @@ -201,58 +219,74 @@ As a quick summary, our current process to create a buildpack package involves: We propose: - Deprecate the `platform.os` field from [package.toml](https://buildpacks.io/docs/reference/config/package-config/). It will be removed after two pack releases with the new feature - When `platform.os` is present in [package.toml](https://buildpacks.io/docs/reference/config/package-config/), throw a warning messages indicating the field will be removed -and `--platform` flag must be used -- When `platform.os` is not present [package.toml](https://buildpacks.io/docs/reference/config/package-config/) and `--platform` flag is not used, throw a warning messages indicating -a new `--platform` flag is available +and `--target` flag must be used +- When `platform.os` is not present [package.toml](https://buildpacks.io/docs/reference/config/package-config/) and `--target` flag is not used, throw a warning messages indicating +a new `--target` flag is available - Keep doing our current process to package a buildpack ### To improve user experience We propose: -- Add a new `--platform` flag with format `[os][/arch][/variant]:[name@version]` to build for a particular target, once the `platform.os` field is removed, +- Add a new `--target` flag with format `[os][/arch][/variant]:[name@version]` to build for a particular target, once the `platform.os` field is removed, this will be the way for end-users to specify the platform for which they want to create single OCI artifact. > **Note** > We want to avoid duplicated information, we expect to have `targets` with all the platform supported by a buildpack -- Add a new boolean `--multi-arch` flag to indicate pack it must create multiples OCI artifacts and combine them with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md), -this flag must be used in conjunction with `--publish` or `--format file` flags and error out when `daemon` is selected. -> **Note** -> I got feedback to avoid creating an Image Index if users are not expecting that, the flag is an acknowledgment for creating the index - - A new folder structure to organize the buildpacks binaries for multi-arch images ```bash +# Option 1 - no variant is required . ├── buildpack.toml -└── {os}-{arch} // optional - └── {variant} // optional - ├── {version-1} // optional - │ └── bin - │ ├── build - │ └── detect - └── {version-2} // optional - └── bin - ├── build - └── detect +└── {os} // optional + └── {arch} // optional + └── bin + ├── build + └── detect + +# Option 2 - variant is required +. +├── buildpack.toml +└── {os} // optional + └── {arch} // optional + └── {variant} // optional + ├── {name@version-1} // optional + │ └── bin + │ ├── build + │ └── detect + └── {name@version-2} // optional + └── bin + ├── build + └── detect ``` > **Note** > For cross-compile buildpacks like Paketo, it looks easy to add a step to their Makefile to compile and separate the binaries following this structure. -Based on the [RFC-0096](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) we replaced the concept of `stacks` in `buildpack.toml` by a `target` which looks like: +In case, this folder structure is not suitable for Buildpack Authors: + +> **Proposal** +> a new `path` attribute is proposed to be included into the `target` section of the `buildpack.toml` to specify +where the binaries are located in the filesystem. + +Based on the [RFC-0096](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) the new `buildpack.toml` schema will look like this: ```toml [[targets]] os = "" arch = "" variant = "" + +# optional +path = "" + [[targets.distributions]] name = "" versions = [""] ``` -- When `--multi-arch` is enabled +- When `more than 1 target is defined` - When `--publish` or `--format file` is specified - - `pack` will read `targets` from `buildpack.toml` - For each `target` an OCI image will be created, following our current process - - `pack` will try to infer a folder structure similar to the one show above, and if it is found, those are the binaries to include in the OCI image + - `pack` will determine the binaries path (based on the `targets.path` in the buildpack.toml or infering a folder structure similar to the one show above) + - `pack` will include the binaries in the buildpack package OCI image - If more than 1 OCI image was created, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them ### Examples @@ -278,16 +312,16 @@ In these cases, the expected output will be similar to: ```bash pack buildpack package --config ./package.toml --publish -Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Warning: A new '--target' flag is available to set the platform for the buildpack package, using 'linux' as default Successfully published package and saved to registry # Or pack buildpack package --config ./package.toml --format file -Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Warning: A new '--target' flag is available to set the platform for the buildpack package, using 'linux' as default Successfully created package and saved to file ``` -We will keep `linux` as the default platform, and we don't expect end-users to change anything else on their side. +We will keep `linux` as the default target platform, and we don't expect end-users to change anything else on their side. In case, Buildpack authors tries to use the new folder structure like: @@ -308,8 +342,7 @@ The current `pack` version creates an OCI artifact copying all the files on it. ![](https://hackmd.io/_uploads/BkGkZ_mla.png) -But I think this buildpack can't be executed because `./bin/build` or `./bin/detect` can't be found. We could add some validations and throw an error, -but I do not see any value on doing that. +But I think this buildpack can't be executed because `./bin/build` or `./bin/detect` can't be found. We could add some validations and throw an error, but I do not see any value on doing that. > **Important** > New folder structure is not useful if Buildpack Authors don't remove stacks and migrate to use targets @@ -319,11 +352,11 @@ Trying to use the new flags ```bash pack buildpack package --config ./package.toml --publish --multi-arch -Error: 'targets' or 'platforms' must be defined when creating a multi-architecture buildpack +Error: 'targets' must be defined when creating a multi-architecture buildpack # Or pack buildpack package --config ./package.toml --format file --multi-arch -Error: 'targets' or 'platforms' must be defined when creating a multi-architecture buildpack +Error: 'targets' must be defined when creating a multi-architecture buildpack ``` In these cases, we don't have enough information to create a multi-arch buildpack, and pack must fail its execution. @@ -331,32 +364,32 @@ In these cases, we don't have enough information to create a multi-arch buildpac A valid way to re-write the command could be: ```bash -pack buildpack package --config ./package.toml --publish --platform linux/arm64 --platform linux/amd64 --multi-arch -A multi-arch buildpack package will be created for platforms: 'linux/amd64', 'linux/arm64' +pack buildpack package --config ./package.toml --publish --target linux/arm64 --target linux/amd64 --multi-arch +A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64' Successfully published package and saved to registry # Or -pack buildpack package --config ./package.toml --format file --platform linux/arm64 --platform linux/amd64 --multi-arch -A multi-arch buildpack package will be created for platforms: 'linux/amd64', 'linux/arm64' +pack buildpack package --config ./package.toml --format file --target linux/arm64 --target linux/amd64 --multi-arch +A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64' Successfully created package and saved to file ``` In these cases, two OCI images, with the same binaries, will be created and pushed into the registry, for each image the configuration file will be created with the correct `os` and `architecture` and an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them -what about creating a single image for a new platform? +what about creating a single image for a new target platform? ```bash -pack buildpack package --config ./package.toml --publish --platform linux/arm64 +pack buildpack package --config ./package.toml --publish --target linux/arm64 Successfully published package and saved to registry # Or -pack buildpack package --config ./package.toml --format file --platform linux/arm64 +pack buildpack package --config ./package.toml --format file --target linux/arm64 Successfully created package and saved to file ``` In these cases, pack will create buildpack package image with the provided binaries and a [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) -with the following platform: +with the following target platform: ```json { @@ -371,12 +404,12 @@ These cases are similar to the previous one, but we can change the warning messa ```bash pack buildpack package --config ./package.toml --publish -Warning: 'platform.os' field in package.toml will be deprecated, use new '--platform' flag to set target platform +Warning: 'platform.os' field in package.toml will be deprecated, use new '--target' flag to set target platform Successfully published package and saved to registry # Or pack buildpack package --config ./package.toml --format file -Warning: 'platform.os' field in package.toml will be deprecated, use new '--platform' flag to set target platform +Warning: 'platform.os' field in package.toml will be deprecated, use new '--target' flag to set target platform Successfully created package and saved to file ``` @@ -413,19 +446,19 @@ this will avoid breaking the current behavior. ```bash pack buildpack package --config ./package.toml --publish Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' -Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Warning: A new '--target' flag is available to set the target platform for the buildpack package, using 'linux' as default Successfully published package and saved to registry # Or pack buildpack package --config ./package.toml --format file Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' -Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Warning: A new '--target' flag is available to set the target platform for the buildpack package, using 'linux' as default Successfully created package and saved to file # Or pack buildpack package --config ./package.toml Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' -Warning: A new '--platform' flag is available to set the target platform for the buildpack package, using 'linux' as default +Warning: A new '--target' flag is available to set the target platform for the buildpack package, using 'linux' as default Successfully created package and saved to docker daemon ``` @@ -438,10 +471,10 @@ The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/ } ``` -On the other hand, when end-users use the new `platform` flag +On the other hand, when end-users use the new `target` flag ```bash -pack buildpack package --config ./package.toml --publish --platform linux/arm64 +pack buildpack package --config ./package.toml --publish --target linux/arm64 Successfully published package and saved to registry ``` @@ -457,15 +490,15 @@ The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/ Trying to create a buildpack package for an unsupported platform should throw an error ```bash -pack buildpack package --config ./package.toml --publish --platform windows/arm64 -Error: platform 'windows/arm64' is not supported in buildpack targets, check your 'buildpack.toml' +pack buildpack package --config ./package.toml --publish --target windows/arm64 +Error: target platform 'windows/arm64' is not supported in buildpack targets, check your 'buildpack.toml' ``` Let's go one more step further, and check what will happen if Buildpack Authors use the new `--multi-arch` flag ```bash pack buildpack package --config ./package.toml --publish --multi-arch -Info: A multi-arch buildpack package will be created for platforms: 'linux/amd64', 'linux/arm64' +Info: A multi-arch buildpack package will be created for targets platforms: 'linux/amd64', 'linux/arm64' Successfully published package and saved to registry ``` @@ -560,10 +593,10 @@ name = "windows" versions = ["10.0.20348.1970"] ``` -If the Buildpack Author wants to create a single buildpack package they will use the `platform` flag +If the Buildpack Author wants to create a single buildpack package they will use the `target` flag ```bash -pack buildpack package --config ./package.toml --publish --platform linux/arm64 +pack buildpack package --config ./package.toml --publish --target linux/arm64 Successfully published package and saved to registry ``` @@ -591,12 +624,12 @@ A fully multi-arch buildpack could be created as follows: ```bash pack buildpack package --config ./package.toml --publish --multi-arch -Info: A multi-arch buildpack package will be created for platforms: 'linux/amd64', 'linux/arm64', 'windows/amd64' +Info: A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64', 'windows/amd64' Successfully published package and saved to registry ``` In this case, three OCI images will be created and pushed into the registry, for each image the configuration file will be -created with the correct platform: `os` and `architecture`, +created with the correct target: `os` and `architecture`, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them, with a content similar to: @@ -639,17 +672,18 @@ with a content similar to: ``` > **Important** -> In this case, each image has different binaries according to the platform and the new folder structure +> In this case, each image has different binaries according to the target platform and the new folder structure ## Builder Similar to how we did it for the `buildpack package`, lets summaries, our current process to create a **Builder**: -- We read the `builder.toml` and fetch the `run.image`, currently we didn't specify the `platform`, **host** `os` is being used. -- We read the `builder.toml` and fetch the `build.image`, currently we didn't specify the `platform`, **host** `os` is being used. +- We read the `builder.toml` and fetch the `build.image`, currently we didn't specify the `platform`, **daemon** `os/arch` is being used. - We create a **base builder** from the `build.image`. -- We read the `os` and `architecture` from the **base builder**, fetch a `lifecycle` image that matches the platform (Note: lifecycle is already publish behind an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md)) -- We add `Buildpacks` and `Extensions` to the **base builder**, because we don't define `architecture` for them, we fetch matching the **base builder** `os` +- We read the `os` and `architecture` from the **base builder** + - Fetch the `run.image`, matching the `os/arch` with the values from the **base builder** + - Fetch the `lifecycle` image that matches the platform (Note: lifecycle is already publish behind an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md)) + - We add `Buildpacks` and `Extensions` to the **base builder** trying to match the **base builder** `os/arch`, in case the `architecture` doesn't match, we fall back to match the **base builder** `os` - More logic and finally the **Builder** image is created ### To keep compatibility @@ -659,7 +693,7 @@ We propose: - When `stacks` is present in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/), throw a warning message indicating the field is deprecated and it will be removed - When `targets` is not present in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/), throw a warning messages indicating - a new `--platform` flag is available + a new `--target` flag is available - Keep doing our current process to create a builder ### To improve user experience @@ -685,7 +719,7 @@ image = "" # MUST point to an Image Index that matches targets image = "" -# Platforms to support with the Builder +# Target platforms to support with the Builder [[targets]] os = "" arch = "" @@ -694,7 +728,7 @@ variant = "" name = "" versions = [""] ``` -- Add a new `--platform` optional flag with format `[os][/arch][/variant]:[name@version]` to create a builder for a particular target, this will help end-users to specify the platform for which they want to create single OCI artifact. +- Add a new `--target` optional flag with format `[os][/arch][/variant]:[name@version]` to create a builder for a particular target, this will help end-users to specify the platform for which they want to create single OCI artifact. - Add a new boolean `--multi-arch` flag to indicate pack it must create multiples OCI artifacts and combine them with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md), this flag must be used in conjunction with `--publish` or `--format file` flags and error out when `daemon` is selected. @@ -739,7 +773,7 @@ In these cases, the expected output will be similar to: ```bash pack builder create --config ./builder.toml Warning: "stack" has been deprecated, prefer "targets" instead: https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md -Warning: A new '--platform' flag is available to set the target platform for the builder, using 'linux/amd64' as default +Warning: A new '--target' flag is available to set the target platform for the builder, using 'linux/amd64' as default Successfully created builder image Tip: Run pack build --builder to use this builder ``` @@ -749,14 +783,14 @@ printed to help end-users to check for new updates, maybe link to a migration gu Trying to use the new flags: ```bash -pack builder create --config ./builder.toml --platform linux/arm64 +pack builder create --config ./builder.toml --target linux/arm64 Warning: "stack" has been deprecated, prefer "targets" instead: https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md Warning: creating a builder for platform "linux/arm64" but "targets" is not defined, update your "builder.toml" to include "targets" Successfully created builder image Tip: Run pack build --builder to use this builder ``` -Pulling operations will be configured to use `linux/arm64` as platform, +Pulling operations will be configured to use `linux/arm64` as target platform, the OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: ```json @@ -770,25 +804,25 @@ What about multi-architecture builders? ```bash pack builder create --config ./builder.toml --multi-arch --publish -Error: 'targets' or 'platforms' must be defined when creating a multi-architecture builder +Error: 'targets' must be defined when creating a multi-architecture builder ``` In this case, pack doesn't have enough information to create a multi-arch builder and fail its execution. -Using `platform` flag: +Using `target` flag: ```bash pack builder create --config ./builder.toml \ --multi-arch \ - --plaform linux/amd64 \ - --platform linux/arm64 \ + --target linux/amd64 \ + --target linux/arm64 \ --publish Successfully created builder image Tip: Run pack build --builder to use this builder ``` In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be -created with the correct platform: `os` and `architecture`, +created with the correct target: `os` and `architecture`, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them #### `Targets` are present in `builder.toml` @@ -825,24 +859,24 @@ Let's suppose we execute the command in a host "linux/amd64" machine ```bash pack builder create --config ./builder.toml -Info: creating a builder for platform "linux/amd64" +Info: creating a builder for target "linux/amd64" Successfully created builder image Tip: Run pack build --builder to use this builder ``` -`platform` flag is not defined, we keep our current behavior and detect the host `os` and `architecture`. Because a +`target` flag is not defined, we keep our current behavior and detect the host `os` and `architecture`. Because a `target` matches the host `platform` it is used to create the builder. Trying to use the new flags: ```bash -pack builder create --config ./builder.toml --platform linux/arm64 -Info: creating a builder for platform "linux/arm64" +pack builder create --config ./builder.toml --target linux/arm64 +Info: creating a builder for target "linux/arm64" Successfully created builder image Tip: Run pack build --builder to use this builder ``` -Pulling operations will be configured to use `linux/arm64` as platform, +Pulling operations will be configured to use `linux/arm64` as target platform, the OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: ```json @@ -856,27 +890,27 @@ What about multi-architecture builders? ```bash pack builder create --config ./builder.toml --multi-arch --publish -Info: A multi-arch builder will be created for platforms: 'linux/amd64', 'linux/arm64' +Info: A multi-arch builder will be created for targets platform: 'linux/amd64', 'linux/arm64' Successfully created builder image Tip: Run pack build --builder to use this builder ``` -Using `platform` flag: +Using `target` flag: ```bash pack builder create --config ./builder.toml \ --multi-arch \ - --plaform linux/amd64 \ - --platform linux/arm64 \ + --target linux/amd64 \ + --target linux/arm64 \ --publish -Info: A multi-arch builder will be created for platforms: 'linux/amd64', 'linux/arm64' +Info: A multi-arch builder will be created for targets platform: 'linux/amd64', 'linux/arm64' Successfully created builder image Tip: Run pack build --builder to use this builder ``` In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be -created with the correct platform: `os` and `architecture`, +created with the correct target platform: `os` and `architecture`, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them @@ -913,7 +947,7 @@ create [image indexes](https://github.com/opencontainers/image-spec/blob/master/ [unresolved-questions]: #unresolved-questions - If I already have an [image indexe](https://github.com/opencontainers/image-spec/blob/master/image-index.md) created and I want to add support for new plaform, how do I do it? -- What are the intermediate images for each platform named/called? +- What are the intermediate images for each target named/called? - What happen if I want to exclude some buildpack for a particular target? - What happen if I want to include the same file or folder for every image, do I have to copy then inside the {os}-{arch} folder? From d4ba0baf2dc9afaf2a39a924b2c3cb5351720d4e Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 10 Nov 2023 16:02:16 -0500 Subject: [PATCH 158/222] fixing some feedback from review, removing --multi-arch flag Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 316 ++++++++++---------- 1 file changed, 158 insertions(+), 158 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index eb5348395..a56145423 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -122,10 +122,10 @@ my-multiarch-buildpack └── detect.bat ``` -Now `pack` will be able to package them separately for each os/arch family, following our [guide](https://buildpacks.io/docs/buildpack-author-guide/package-a-buildpack/) -we will need to create a `package.toml` file. +Now `pack` will be able to package them separately for each os/arch family. -The `package.toml` file will be similar to the one describe in our documentation: +Following our [guide](https://buildpacks.io/docs/buildpack-author-guide/package-a-buildpack/) we will need to create a +`package.toml` let's suppose it looks like this: ```toml [buildpack] @@ -134,7 +134,7 @@ uri = "examples/my-multiarch-buildpack" uri = "my-multiarch-buildpack.tgz" ``` -But in this case, we actually **remove** the [platform](https://buildpacks.io/docs/reference/config/package-config/) section because it will be taken from the `buildpack.toml`. +In this case we **remove** the [platform](https://buildpacks.io/docs/reference/config/package-config/) section because it will be taken from the `buildpack.toml`. Packaging a multi-arch buildpack will require the output to be **publish** to a registry or **saved on disk** in OCI layout format. @@ -147,13 +147,13 @@ pack buildpack package my-buildpack.cnb --config ./package.toml --format file > **Important** > pack will determine a multi-arch buildpack package is being created because there are more than one target defined. -In these cases each `target` entry corresponds to a different buildpack image that is exported into an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) +Each `target` entry corresponds to a different buildpack image that is exported into an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) ### Package a multi-arch Builder In case of packing a **Builder**, we assume the following premises: -1. Buildpack authors updated their `builder.toml` to include the new `targets` fields defined in this RFC. +1. Buildpack authors updated their `builder.toml` to include the new `targets` fields defined in this [RFC](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md). 2. Multi-architecture `build`, `run` images and `buildpacks` are available for baking into the **Builder**. A sample `builder.toml` file looks like: @@ -193,11 +193,13 @@ As we can see, the proposal is based on the assumption that the `run-image`, `bu in the builder are **multi-arch artifacts**, and we can reach them by reading an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) +Packaging a multi-arch builder will require the output to be **publish** to a registry. + ```bash pack builder create my-jammy-builder --config ./builder.toml --publish ``` > **Important** -> Similar to the `buildpack package`, pack will determine a multi-arch builder must be created based on the multiple targets defined. +> Similar to the `buildpack package,` pack will determine a multi-arch builder must be created based on the multiple targets defined. In this case `pack` will follow the builder creation process for **each provided target**, pulling the correct (based on os/arch) buildpacks, build and run images and creating different builders images that are @@ -210,7 +212,7 @@ exported and combined into an [image index](https://github.com/opencontainers/im As a quick summary, our current process to create a buildpack package involves: -- The end-users defined `os` for the OCI image using the [package.toml](https://buildpacks.io/docs/reference/config/package-config/) when they want to export a buildpack package. +- The end-users defined `os` for the OCI image using the [package.toml](https://buildpacks.io/docs/reference/config/package-config/). - The only values allowed are `linux` and `windows` and by default when is not present, `linux` is being used. - When exporting to daemon, the `docker.OSType` must be equal to `platform.os` @@ -232,6 +234,8 @@ this will be the way for end-users to specify the platform for which they want t > **Note** > We want to avoid duplicated information, we expect to have `targets` with all the platform supported by a buildpack +- Add `targets` section to `buildpack.toml` to help Buildpack Authors to include support for new platforms without having to update their `pack buildpack package` command in their CI/CD pipelines + - A new folder structure to organize the buildpacks binaries for multi-arch images ```bash # Option 1 - no variant is required @@ -259,7 +263,8 @@ this will be the way for end-users to specify the platform for which they want t └── detect ``` > **Note** -> For cross-compile buildpacks like Paketo, it looks easy to add a step to their Makefile to compile and separate the binaries following this structure. +> For cross-compile buildpacks like Paketo, it looks easy to add a step to their Makefile to compile and separate the binaries following this structure. It is important to mention +> that the final buildpack image will not change, this will only change the buildpack structure from `pack` perspective In case, this folder structure is not suitable for Buildpack Authors: @@ -288,17 +293,20 @@ versions = [""] - `pack` will determine the binaries path (based on the `targets.path` in the buildpack.toml or infering a folder structure similar to the one show above) - `pack` will include the binaries in the buildpack package OCI image - If more than 1 OCI image was created, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them + - When `--daemon` is specified + - `pack` can keep using `docker.OSType` to determine the target `os` and probably can do some validations it the `os` is valid target ### Examples Let's use some examples to explain the expected behavior in different use cases -#### Buildpacks authors do not use targets - -##### `platform.os` is not present at `package.toml` +#### Buildpacks authors do not use targets AND `platform.os` is not present at `package.toml` This seems to be the case for [Paketo Buildpacks](https://github.com/paketo-buildpacks/java/blob/main/package.toml) -or [Heroku](https://github.com/heroku/buildpacks-jvm/blob/main/meta-buildpacks/java/package.toml). A simplified version of their folder structures is: +or [Heroku](https://github.com/heroku/buildpacks-jvm/blob/main/meta-buildpacks/java/package.toml), and it represents how `pack` +will work for most the users when new behavior is implemented + +A simplified version of Buildpack Authors folder structures is: ```bash ├── bin @@ -308,7 +316,8 @@ or [Heroku](https://github.com/heroku/buildpacks-jvm/blob/main/meta-buildpacks/j └── package.toml ``` -In these cases, the expected output will be similar to: +In these cases: We expect `pack` to keep doing what is doing today, but with the warning messages we mentioned above to +let end users know things are changing. ```bash pack buildpack package --config ./package.toml --publish @@ -319,104 +328,112 @@ Successfully published package and saved to registry pack buildpack package --config ./package.toml --format file Warning: A new '--target' flag is available to set the platform for the buildpack package, using 'linux' as default Successfully created package and saved to file -``` - -We will keep `linux` as the default target platform, and we don't expect end-users to change anything else on their side. - -In case, Buildpack authors tries to use the new folder structure like: + ``` +**Output**: pack will create a buildpack package image (as it is doing it today) with the provided binaries and a +[configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) with the following target +platform: -```bash -├── buildpack.toml -├── linux-amd64 -│ └── bin -│ ├── build -│ └── detect -├── linux-arm64 -│ └── bin -│ ├── build -│ └── detect -└── package.toml +```json +{ + "architecture": "", + "os": "linux" +} ``` -The current `pack` version creates an OCI artifact copying all the files on it. - -![](https://hackmd.io/_uploads/BkGkZ_mla.png) - -But I think this buildpack can't be executed because `./bin/build` or `./bin/detect` can't be found. We could add some validations and throw an error, but I do not see any value on doing that. - -> **Important** -> New folder structure is not useful if Buildpack Authors don't remove stacks and migrate to use targets -> - -Trying to use the new flags +After checking the **warning** messages, some end users must feel curious, and the try to use the new `--target` flag. ```bash -pack buildpack package --config ./package.toml --publish --multi-arch -Error: 'targets' must be defined when creating a multi-architecture buildpack +pack buildpack package --config ./package.toml --publish --target linux/arm64 +Successfully published package and saved to registry # Or -pack buildpack package --config ./package.toml --format file --multi-arch -Error: 'targets' must be defined when creating a multi-architecture buildpack +pack buildpack package --config ./package.toml --format file --target linux/arm64 +Successfully created package and saved to file ``` -In these cases, we don't have enough information to create a multi-arch buildpack, and pack must fail its execution. +**Output**: In these cases, pack will create buildpack a package image with the provided binaries and a +[configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) with the following target +platform: -A valid way to re-write the command could be: +```json +{ + "architecture": "arm64", + "os": "linux" +} +``` + +what about creating a multi-arch image for several target platforms? ```bash -pack buildpack package --config ./package.toml --publish --target linux/arm64 --target linux/amd64 --multi-arch +pack buildpack package --config ./package.toml --publish --target linux/arm64 --target linux/amd64 A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64' Successfully published package and saved to registry # Or -pack buildpack package --config ./package.toml --format file --target linux/arm64 --target linux/amd64 --multi-arch +pack buildpack package --config ./package.toml --format file --target linux/arm64 --target linux/amd64 A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64' Successfully created package and saved to file ``` -In these cases, two OCI images, with the same binaries, will be created and pushed into the registry, for each image the configuration file will be -created with the correct `os` and `architecture` and an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them +**Output**: two OCI images, with the same binaries, will be created and pushed into the registry, for each image the +configuration file will be created with the correct `os` and `architecture` and an +[image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them +using the `` name provided. + +#### Buildpacks authors do not use targets AND `platform.os` is present at `package.toml` + +Let's suppose the `package.toml` has the following: -what about creating a single image for a new target platform? +```toml +[buildpack] +uri = "" + +[platform] +os = "linux" +``` +These cases are similar to the previous one, but we can change the warning message. ```bash -pack buildpack package --config ./package.toml --publish --target linux/arm64 +pack buildpack package --config ./package.toml --publish +Warning: 'platform.os' field in package.toml will be deprecated, use new '--target' flag or `targets` field in buildpack.toml to set the platform. Successfully published package and saved to registry # Or -pack buildpack package --config ./package.toml --format file --target linux/arm64 +pack buildpack package --config ./package.toml --format file +Warning: 'platform.os' field in package.toml will be deprecated, use new '--target' flag or `targets` field in buildpack.toml to set the platform. Successfully created package and saved to file ``` - -In these cases, pack will create buildpack package image with the provided binaries and a [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) -with the following target platform: +**Output**: The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: ```json { - "architecture": "arm64", + "architecture": "", "os": "linux" } ``` -##### `platform.os` is present at `package.toml` - -These cases are similar to the previous one, but we can change the warning message. +Trying to use `--target` flag with `platform.os` field at the same time should throw an error, in this way, the end-user will need to update +their `package.toml` ```bash -pack buildpack package --config ./package.toml --publish -Warning: 'platform.os' field in package.toml will be deprecated, use new '--target' flag to set target platform -Successfully published package and saved to registry - +pack buildpack package --config ./package.toml --publish --target linux/arm64 # Or -pack buildpack package --config ./package.toml --format file -Warning: 'platform.os' field in package.toml will be deprecated, use new '--target' flag to set target platform -Successfully created package and saved to file +pack buildpack package --config ./package.toml --format file --target linux/arm64 + +Error: 'platform.os' and '--target' flag can not be used in conjunction, please remove 'platform.os' from package.toml, use new '--target' flag + or `targets` field in buildpack.toml to set the platform ``` #### Buildpacks authors use targets +> **Important** +> `pack` considers the use of `targets` as an acknowledgement of expecting a multi-arch images as output. Also, we expect +> `platform.os` do not be present in `buildpack.toml` + We can divide the problem in two main scenarios: Buildpack authors use or not use the new folder structure. +##### New folder structure is not use + Let's start with the first one, which is the natural path for Buildpack Authors to migrate from `stacks` to `targets`. Let's suppose a buildpack folder structure like: ```bash @@ -437,73 +454,25 @@ arch = "amd64" [[targets]] os = "linux" arch = "arm64" -``` -How could Buildpack Authors set which `os/arch` to use?, currently `pack` doesn't care about `targets` and -it sets `linux` as the `os` in the final OCI image. We can keep this behavior, and show warning messages to end-users, -this will avoid breaking the current behavior. +# Stacks (deprecated) the buildpack will work with +[[stacks]] +id = "*" +``` ```bash pack buildpack package --config ./package.toml --publish -Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' -Warning: A new '--target' flag is available to set the target platform for the buildpack package, using 'linux' as default +A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64' Successfully published package and saved to registry # Or pack buildpack package --config ./package.toml --format file -Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' -Warning: A new '--target' flag is available to set the target platform for the buildpack package, using 'linux' as default +A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64' Successfully created package and saved to file - -# Or -pack buildpack package --config ./package.toml -Warning: Multiple targets 'linux/amd64', 'linux/arm64' are present at 'buildpack.toml' -Warning: A new '--target' flag is available to set the target platform for the buildpack package, using 'linux' as default -Successfully created package and saved to docker daemon -``` - -The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: - -```json -{ - "architecture": "", - "os": "linux" -} ``` -On the other hand, when end-users use the new `target` flag - -```bash -pack buildpack package --config ./package.toml --publish --target linux/arm64 -Successfully published package and saved to registry -``` - -The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: - -```json -{ - "architecture": "amd64", - "os": "linux" -} -``` - -Trying to create a buildpack package for an unsupported platform should throw an error - -```bash -pack buildpack package --config ./package.toml --publish --target windows/arm64 -Error: target platform 'windows/arm64' is not supported in buildpack targets, check your 'buildpack.toml' -``` - -Let's go one more step further, and check what will happen if Buildpack Authors use the new `--multi-arch` flag - -```bash -pack buildpack package --config ./package.toml --publish --multi-arch -Info: A multi-arch buildpack package will be created for targets platforms: 'linux/amd64', 'linux/arm64' -Successfully published package and saved to registry -``` - -In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be -created with the correct `os` and `architecture` and +**Output**: In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be +created with the correct `os` and `architecture` and an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them, with a content similar to: @@ -534,43 +503,70 @@ with a content similar to: "schemaVersion": 2 } ``` -> **Important** -> In this case, the layer blob for the OCI image will have the same bin/build and bin/detect binaries. This could be helpful -> for bash buildpacks for example. -There is a downside for this case, how can we do it if we have windows and linux scripts? -One way to do it could be just copying the .bat scripts along with the linux scripts, something like: +In case of targeting the daemon, pack will match **daemon os/arch** with the **targets os/arch**, for example when running +on a `linux/arm64` machine. ```bash -├── bin -│ ├── build -│ ├── build.bat -│ ├── detect -│ └── detect.bat -├── buildpack.toml -└── package.toml +pack buildpack package --config ./package.toml +Successfully created package and saved to docker daemon ``` -But we will end up distributing windows and linux scripts together. +**Output**: In these cases, pack will create buildpack a package image with the provided binaries and a +[configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) with the following target +platform: -Finally, let's check some examples for the second scenario, when Buildpack Authors migrated from `stacks` to `targets` and -they want to take advantage of the new multi-architecture capabilities, let's use our original folder structure: +```json +{ + "architecture": "arm64", + "os": "linux" +} +``` +But, if we execute the same command on a **windows/amd64** machine, platform doesn't match and an error is thrown ```bash +pack buildpack package --config ./package.toml +Error: daemon platform 'windows/amd64' does not match target platforms: 'linux/amd64', 'linux/arm64' +``` +On the other hand, when end-users use the new `--target` flag + +```bash +pack buildpack package --config ./package.toml --publish --target linux/arm64 +Successfully published package and saved to registry +``` + +**Output**: The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: + +```json +{ + "architecture": "amd64", + "os": "linux" +} +``` + +##### New folder structure is use + +Finally, let's check some examples for the second scenario, when Buildpack Authors want to take advantage of the new +multi-architecture capabilities, let's use our original folder structure: + +```bash +. ├── buildpack.toml -├── linux-amd64 -│ └── bin -│ ├── build -│ └── detect -├── linux-arm64 -│ └── bin -│ ├── build -│ └── detect -└── windows-amd64 - └── 10.0.20348.1970 - └── bin - ├── build.bat - └── detect.bat +├── linux +│ ├── amd64 +│ │ └── bin +│ │ ├── build +│ │ └── detect +│ └── arm64 +│ └── bin +│ ├── build +│ └── detect +└── windows + └── amd64 + └── windows@10.0.20348.1970 + └── bin + ├── build.bat + └── detect.bat ``` And a `buildpack.toml` with the following `targets` defined: @@ -591,6 +587,10 @@ arch = "amd64" [[targets.distributions]] name = "windows" versions = ["10.0.20348.1970"] + +# Stacks (deprecated) the buildpack will work with +[[stacks]] +id = "*" ``` If the Buildpack Author wants to create a single buildpack package they will use the `target` flag @@ -600,18 +600,18 @@ pack buildpack package --config ./package.toml --publish --target li Successfully published package and saved to registry ``` -In this case, `pack` will determine the folder `linux-arm64` exists, and that must be the root folder for the buildpack. +In this case, `pack` will find the folder `linux/arm64`, and it must be the root folder for the buildpack. `pack` will copy `buildpack.toml` from the root and include it, something like: ```bash -linux-arm64 +linux/arm64 ├── bin │ ├── build │ └── detect └── buildpack.toml // Must be copy on the fly ``` -Similar to previous cases, the OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: +**Output**: The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: ```json { @@ -620,15 +620,15 @@ Similar to previous cases, the OCI Image [configuration](https://github.com/open } ``` -A fully multi-arch buildpack could be created as follows: +A fully multi-arch buildpack will be created automatically, because we have more than one target defined. ```bash -pack buildpack package --config ./package.toml --publish --multi-arch +pack buildpack package --config ./package.toml --publish Info: A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64', 'windows/amd64' Successfully published package and saved to registry ``` -In this case, three OCI images will be created and pushed into the registry, for each image the configuration file will be +**Output**: In this case, three OCI images will be created and pushed into the registry, for each image the configuration file will be created with the correct target: `os` and `architecture`, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them, with a content similar to: From 614052bb9729e4726301d7e6e9c69f7ebe619249 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 10 Nov 2023 16:32:08 -0500 Subject: [PATCH 159/222] removing multi-arch flag Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 256 ++++++++++++-------- 1 file changed, 157 insertions(+), 99 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index a56145423..420c7f5d6 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -222,55 +222,50 @@ We propose: - Deprecate the `platform.os` field from [package.toml](https://buildpacks.io/docs/reference/config/package-config/). It will be removed after two pack releases with the new feature - When `platform.os` is present in [package.toml](https://buildpacks.io/docs/reference/config/package-config/), throw a warning messages indicating the field will be removed and `--target` flag must be used -- When `platform.os` is not present [package.toml](https://buildpacks.io/docs/reference/config/package-config/) and `--target` flag is not used, throw a warning messages indicating +- When `platform.os` is not present in [package.toml](https://buildpacks.io/docs/reference/config/package-config/) and `--target` flag is not used, throw a warning messages indicating a new `--target` flag is available - Keep doing our current process to package a buildpack ### To improve user experience We propose: -- Add a new `--target` flag with format `[os][/arch][/variant]:[name@version]` to build for a particular target, once the `platform.os` field is removed, +- Add a new `--target` flag with the follwing format `[os][/arch][/variant]:[name@version]` to build for a particular target, once the `platform.os` field is removed, this will be the way for end-users to specify the platform for which they want to create single OCI artifact. -> **Note** -> We want to avoid duplicated information, we expect to have `targets` with all the platform supported by a buildpack - Add `targets` section to `buildpack.toml` to help Buildpack Authors to include support for new platforms without having to update their `pack buildpack package` command in their CI/CD pipelines -- A new folder structure to organize the buildpacks binaries for multi-arch images +- A new folder structure to organize the buildpacks binaries for multi-arch images, similar to this one: ```bash # Option 1 - no variant is required . -├── buildpack.toml -└── {os} // optional - └── {arch} // optional +├── buildpack.toml // mandatory +└── {os} // optional + └── {arch} // optional └── bin - ├── build - └── detect + ├── build // platform dependant binary (mandatory) + └── detect // platform dependant binary (mandatory) # Option 2 - variant is required . -├── buildpack.toml +├── buildpack.toml // mandatory └── {os} // optional └── {arch} // optional └── {variant} // optional ├── {name@version-1} // optional │ └── bin - │ ├── build - │ └── detect + │ ├── build // platform dependant binary (mandatory) + │ └── detect // platform dependant binary (mandatory) └── {name@version-2} // optional └── bin - ├── build - └── detect + ├── build // platform dependant binary (mandatory) + └── detect // platform dependant binary (mandatory) ``` > **Note** > For cross-compile buildpacks like Paketo, it looks easy to add a step to their Makefile to compile and separate the binaries following this structure. It is important to mention > that the final buildpack image will not change, this will only change the buildpack structure from `pack` perspective -In case, this folder structure is not suitable for Buildpack Authors: - -> **Proposal** -> a new `path` attribute is proposed to be included into the `target` section of the `buildpack.toml` to specify -where the binaries are located in the filesystem. +In case, this folder structure is not suitable for Buildpack Authors, **we propose** a new `path` attribute to be included +into the `target` section of the `buildpack.toml` to specify where the binaries are located in the filesystem. Based on the [RFC-0096](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) the new `buildpack.toml` schema will look like this: @@ -362,6 +357,10 @@ platform: } ``` +> **Important** +> Pack will assume the binaries are appropriate for the given target platform, what the flag is doing is expose a mechanism +> to update the metadata present in the OCI config file + what about creating a multi-arch image for several target platforms? ```bash @@ -378,7 +377,36 @@ Successfully created package and saved to file **Output**: two OCI images, with the same binaries, will be created and pushed into the registry, for each image the configuration file will be created with the correct `os` and `architecture` and an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them -using the `` name provided. +using the `` name provided. The content of the [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) +will be similar to: + +```json + +{ + "manifests": [ + { + "digest": "sha256:b492494d8e0113c4ad3fe4528a4b5ff89faa5331f7d52c5c138196f69ce176a6", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": { + "architecture": "amd64", + "os": "linux" + }, + "size": 424 + }, + { + "digest": "sha256:2589fe6bcf90466564741ae0d8309d1323f33b6ec8a5d401a62d0b256bcc3c37", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": { + "architecture": "arm", + "os": "linux" + }, + "size": 424 + } + ], + "mediaType": "application/vnd.oci.image.index.v1+json", + "schemaVersion": 2 +} +``` #### Buildpacks authors do not use targets AND `platform.os` is present at `package.toml` @@ -391,7 +419,7 @@ uri = "" [platform] os = "linux" ``` -These cases are similar to the previous one, but we can change the warning message. +These cases are similar to the previous one, but we the warning message will be changed. ```bash pack buildpack package --config ./package.toml --publish @@ -434,7 +462,8 @@ We can divide the problem in two main scenarios: Buildpack authors use or not us ##### New folder structure is not use -Let's start with the first one, which is the natural path for Buildpack Authors to migrate from `stacks` to `targets`. Let's suppose a buildpack folder structure like: +Let's start with the first one, which is the natural path for Buildpack Authors that are using `bash` buildpacks. +Let's suppose a buildpack folder structure like: ```bash ├── bin @@ -504,6 +533,22 @@ with a content similar to: } ``` +On the other hand, when end-users use the new `--target` flag, they can create a single OCI artifact + +```bash +pack buildpack package --config ./package.toml --publish --target linux/arm64 +Successfully published package and saved to registry +``` + +**Output**: The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: + +```json +{ + "architecture": "amd64", + "os": "linux" +} +``` + In case of targeting the daemon, pack will match **daemon os/arch** with the **targets os/arch**, for example when running on a `linux/arm64` machine. @@ -512,7 +557,7 @@ pack buildpack package --config ./package.toml Successfully created package and saved to docker daemon ``` -**Output**: In these cases, pack will create buildpack a package image with the provided binaries and a +**Output**: pack will create a buildpack package image with the provided binaries and a [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) with the following target platform: @@ -523,26 +568,12 @@ platform: } ``` -But, if we execute the same command on a **windows/amd64** machine, platform doesn't match and an error is thrown +But, if we execute the same command on a **windows/amd64** machine, the `buildpack.toml` doesn't contain any `target` that +matches the **daemon os/arch**, an error must be thrown ```bash pack buildpack package --config ./package.toml Error: daemon platform 'windows/amd64' does not match target platforms: 'linux/amd64', 'linux/arm64' ``` -On the other hand, when end-users use the new `--target` flag - -```bash -pack buildpack package --config ./package.toml --publish --target linux/arm64 -Successfully published package and saved to registry -``` - -**Output**: The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: - -```json -{ - "architecture": "amd64", - "os": "linux" -} -``` ##### New folder structure is use @@ -593,40 +624,14 @@ versions = ["10.0.20348.1970"] id = "*" ``` -If the Buildpack Author wants to create a single buildpack package they will use the `target` flag - -```bash -pack buildpack package --config ./package.toml --publish --target linux/arm64 -Successfully published package and saved to registry -``` - -In this case, `pack` will find the folder `linux/arm64`, and it must be the root folder for the buildpack. -`pack` will copy `buildpack.toml` from the root and include it, something like: - -```bash -linux/arm64 -├── bin -│ ├── build -│ └── detect -└── buildpack.toml // Must be copy on the fly -``` - -**Output**: The OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: - -```json -{ - "architecture": "arm64", - "os": "linux" -} -``` - -A fully multi-arch buildpack will be created automatically, because we have more than one target defined. - +When Buildpack Authors want to create a multi-arch images, they can execute the following command: ```bash pack buildpack package --config ./package.toml --publish Info: A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64', 'windows/amd64' Successfully published package and saved to registry ``` +A fully multi-arch buildpack will be created automatically, because we have more than one target defined +in the `buildpack.toml` **Output**: In this case, three OCI images will be created and pushed into the registry, for each image the configuration file will be created with the correct target: `os` and `architecture`, @@ -671,9 +676,63 @@ with a content similar to: } ``` -> **Important** -> In this case, each image has different binaries according to the target platform and the new folder structure - +If the Buildpack Author wants to create a single buildpack package they will use the `target` flag, similar to our previous +examples. + +### Important considerations for share files in the new folder structure + +Let's suppose we have a multi-arch buildpack **A** with local folder structure like this: + +```bash +. +├── LICENSE +├── NOTICE +├── README.md +├── buildpack.toml +├── resources +│ ├── config.properties +└── linux + ├── amd64 + │ └── bin + │ ├── build + │ ├── detect + │ └── foo + └── arm64 + └── bin + ├── build + ├── detect + └── bar +``` + +As we already discussed in this RFC, we are expecting to create two OCI images artifacts, combined with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). +But what happen with the layers on those OCI images and the files that are shared? + +If we update the buildpack package creation process as follows: + +- If `A` is a multi-arch buildpack +- If `Platform` is a target defined for the buildpack A that follows a folder structure like `{os}/{arch}[/{variant}/{name@version-1}]/bin` + +$$BuildpackPackageLayers(A, Platform) = ShareLayers(A) + PlatformLayers(A, Platform)$$ + +Where + +- **ShareLayers**: Will be a layer created with the content of the root folder filtering out any `{os}` folder +- **PlatformLayers**: Will be a layer created with the content of the `Platform` folder given + +With our previous example, it means + +$$ ShareLayers(A) = \{\text{LICENSE},\text{NOTICE},\text{README.md},\text{buildpack.toml},\text{resources/config.properties}\} $$ + +$$ PlatformLayers(A,\text{linux,amd64}) = \{\text{bin/build},\text{bin/detect},\text{bin/foo}\} $$ + +$$ PlatformLayers(A,\text{linux,arm64}) = \{\text{bin/build},\text{bin/detect},\text{bin/bar}\} $$ + +$$ BuildpackPackageLayers(A,\text{linux,amd64}) = ShareLayers(A) + PlatformLayers(A,\text{linux,amd64}) $$ + +$$ BuildpackPackageLayers(A,\text{linux,arm64}) = ShareLayers(A) + PlatformLayers(A,\text{linux,arm64}) $$ + +This means the shared layer can be reused in each intermediate image and improve the performance when it contains big files + ## Builder Similar to how we did it for the `buildpack package`, lets summaries, our current process to create a **Builder**: @@ -700,7 +759,11 @@ it will be removed We propose: -- Add `targets` section to the `builder.toml` schema, this will keep consistency for end-users to understand how to define multi-architecture. The new schema will be +- Add `targets` section to the `builder.toml` schema, this will keep consistency for end-users to understand how to +define multi-architecture. Adding more than one target to the `builder.toml` will be considered by `pack` as an +acknowledgement of the desired to generate multi-arch [Builders](https://buildpacks.io/docs/concepts/components/builder/). + +The new schema will be similar to: ```toml # Buildpacks to include in builder, @@ -728,9 +791,8 @@ variant = "" name = "" versions = [""] ``` -- Add a new `--target` optional flag with format `[os][/arch][/variant]:[name@version]` to create a builder for a particular target, this will help end-users to specify the platform for which they want to create single OCI artifact. -- Add a new boolean `--multi-arch` flag to indicate pack it must create multiples OCI artifacts and combine them with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md), - this flag must be used in conjunction with `--publish` or `--format file` flags and error out when `daemon` is selected. +- Add a new `--target` optional flag with format `[os][/arch][/variant]:[name@version]` to create a builder for a +particular target, this will help end-users to specify the platform for which they want to create single OCI artifact. ### Examples @@ -738,8 +800,9 @@ Let's use some examples to explain the expected behavior in different use cases #### `Targets` are not present in `builder.toml` -This is probably the case for most of the Buildpack Authors, for example [Paketo](https://github.com/paketo-buildpacks/builder-jammy-tiny/blob/main/builder.toml), lets suppose a -`buildpack.toml` like: +This is probably the case for most of the Buildpack Authors, for example +[Paketo](https://github.com/paketo-buildpacks/builder-jammy-tiny/blob/main/builder.toml), lets suppose a`buildpack.toml` +like: ```toml # Buildpacks to include in builder @@ -790,7 +853,7 @@ Successfully created builder image Tip: Run pack build --builder to use this builder ``` -Pulling operations will be configured to use `linux/arm64` as target platform, +**Output**: Pulling operations will be configured to use `linux/arm64` as target platform, the OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) file will have: ```json @@ -802,18 +865,10 @@ the OCI Image [configuration](https://github.com/opencontainers/image-spec/blob/ What about multi-architecture builders? -```bash -pack builder create --config ./builder.toml --multi-arch --publish -Error: 'targets' must be defined when creating a multi-architecture builder -``` - -In this case, pack doesn't have enough information to create a multi-arch builder and fail its execution. - Using `target` flag: ```bash pack builder create --config ./builder.toml \ - --multi-arch \ --target linux/amd64 \ --target linux/arm64 \ --publish @@ -821,9 +876,9 @@ Successfully created builder image Tip: Run pack build --builder to use this builder ``` -In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be -created with the correct target: `os` and `architecture`, -an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them +**Output**: two OCI images will be created and pushed into the registry, for each image the configuration file will be +created with the correct target: `os` and `architecture`, an +[image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them #### `Targets` are present in `builder.toml` @@ -855,7 +910,7 @@ os = "linux" arch = "arm64" ``` -Let's suppose we execute the command in a host "linux/amd64" machine +Let's suppose we execute the command against a daemon running in a `linux/amd64` machine ```bash pack builder create --config ./builder.toml @@ -864,14 +919,18 @@ Successfully created builder image Tip: Run pack build --builder to use this builder ``` -`target` flag is not defined, we keep our current behavior and detect the host `os` and `architecture`. Because a -`target` matches the host `platform` it is used to create the builder. +**Output**: We keep our current behavior and detect the `os` and `architecture` from the daemon. Because there is `target` +that matches the daemon `os/arch` the builder is being built. -Trying to use the new flags: +Using `--target` flag against the daemon with a different platform + +TODO check if this is possible! + What about multi-architecture builders? ```bash -pack builder create --config ./builder.toml --multi-arch --publish +pack builder create --config ./builder.toml --publish Info: A multi-arch builder will be created for targets platform: 'linux/amd64', 'linux/arm64' Successfully created builder image Tip: Run pack build --builder to use this builder ``` - Using `target` flag: ```bash pack builder create --config ./builder.toml \ - --multi-arch \ --target linux/amd64 \ --target linux/arm64 \ --publish @@ -909,7 +967,7 @@ Successfully created builder image Tip: Run pack build --builder to use this builder ``` -In this case, two OCI images will be created and pushed into the registry, for each image the configuration file will be +**Output** In both cases, two OCI images will be created and pushed into the registry, for each image the configuration file will be created with the correct target platform: `os` and `architecture`, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them From 7d017d8dd6b1d08eedbf4e35a5fffda682e18c15 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 24 Nov 2023 17:55:54 -0500 Subject: [PATCH 160/222] fixing some format with the formulas Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index 420c7f5d6..7e5eb6f0e 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -709,8 +709,8 @@ But what happen with the layers on those OCI images and the files that are share If we update the buildpack package creation process as follows: -- If `A` is a multi-arch buildpack -- If `Platform` is a target defined for the buildpack A that follows a folder structure like `{os}/{arch}[/{variant}/{name@version-1}]/bin` +- `A` is a multi-arch buildpack +- `Platform` is a target defined for the buildpack A that follows a folder structure like `{os}/{arch}[/{variant}/{name@version-1}]/bin` $$BuildpackPackageLayers(A, Platform) = ShareLayers(A) + PlatformLayers(A, Platform)$$ From cf6879e9cb6a65f2bcc723647fc9cf651f42bd34 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 27 Nov 2023 16:08:16 -0500 Subject: [PATCH 161/222] updating shared folders definition Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index 7e5eb6f0e..15719dbff 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -679,7 +679,7 @@ with a content similar to: If the Buildpack Author wants to create a single buildpack package they will use the `target` flag, similar to our previous examples. -### Important considerations for share files in the new folder structure +### Important considerations for sharing files with the new folder structure Let's suppose we have a multi-arch buildpack **A** with local folder structure like this: @@ -704,22 +704,26 @@ Let's suppose we have a multi-arch buildpack **A** with local folder structure l └── bar ``` -As we already discussed in this RFC, we are expecting to create two OCI images artifacts, combined with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). +As we already discussed in this RFC, we are expecting to create two OCI images artifacts and combine with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). But what happen with the layers on those OCI images and the files that are shared? -If we update the buildpack package creation process as follows: +Let's suppose we have the following: + +Given - `A` is a multi-arch buildpack -- `Platform` is a target defined for the buildpack A that follows a folder structure like `{os}/{arch}[/{variant}/{name@version-1}]/bin` +- `Platform` is a target defined for `A` that follows a structure like `{os}/{arch}[/{variant}/{name@version-1}]/bin` + +We define the buildpack package layers as: $$BuildpackPackageLayers(A, Platform) = ShareLayers(A) + PlatformLayers(A, Platform)$$ Where -- **ShareLayers**: Will be a layer created with the content of the root folder filtering out any `{os}` folder -- **PlatformLayers**: Will be a layer created with the content of the `Platform` folder given +- **ShareLayers**: Will be a layer created with the content of the root folder filtering out all the `{os}` folders +- **PlatformLayers**: Will be a layer created with the `bin` folder content of the given `Platform` -With our previous example, it means +Using our previous example: $$ ShareLayers(A) = \{\text{LICENSE},\text{NOTICE},\text{README.md},\text{buildpack.toml},\text{resources/config.properties}\} $$ From 93a2d3a6969021b4ea5ce698b5ee6be6a0713640 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 30 Nov 2023 11:01:39 -0500 Subject: [PATCH 162/222] fixing the shared layers concept, because we can't do it based on the distribution spec Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 53 ++++++++++----------- 1 file changed, 26 insertions(+), 27 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index 15719dbff..54c6d9fc2 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -689,6 +689,8 @@ Let's suppose we have a multi-arch buildpack **A** with local folder structure l ├── NOTICE ├── README.md ├── buildpack.toml +├── package.toml +├── some-file // a file with a big size for example ├── resources │ ├── config.properties └── linux @@ -704,38 +706,35 @@ Let's suppose we have a multi-arch buildpack **A** with local folder structure l └── bar ``` -As we already discussed in this RFC, we are expecting to create two OCI images artifacts and combine with an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). -But what happen with the layers on those OCI images and the files that are shared? +As we already discussed in this RFC, we are expecting to create two OCI images artifacts and combine with an +[image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). The current [distribution spec](https://github.com/buildpacks/spec/blob/main/distribution.md#buildpackage) +defines: -Let's suppose we have the following: - -Given - -- `A` is a multi-arch buildpack -- `Platform` is a target defined for `A` that follows a structure like `{os}/{arch}[/{variant}/{name@version-1}]/bin` - -We define the buildpack package layers as: - -$$BuildpackPackageLayers(A, Platform) = ShareLayers(A) + PlatformLayers(A, Platform)$$ - -Where - -- **ShareLayers**: Will be a layer created with the content of the root folder filtering out all the `{os}` folders -- **PlatformLayers**: Will be a layer created with the `bin` folder content of the given `Platform` - -Using our previous example: - -$$ ShareLayers(A) = \{\text{LICENSE},\text{NOTICE},\text{README.md},\text{buildpack.toml},\text{resources/config.properties}\} $$ - -$$ PlatformLayers(A,\text{linux,amd64}) = \{\text{bin/build},\text{bin/detect},\text{bin/foo}\} $$ +``` +Each buildpack layer blob MUST contain a single buildpack at the following file path: -$$ PlatformLayers(A,\text{linux,arm64}) = \{\text{bin/build},\text{bin/detect},\text{bin/bar}\} $$ +/cnb/buildpacks/// +``` -$$ BuildpackPackageLayers(A,\text{linux,amd64}) = ShareLayers(A) + PlatformLayers(A,\text{linux,amd64}) $$ +This means, we will end up duplicating the big file in each OCI images, for example our `linux/amd64` buildpack package +will have 1 layer with the following content under `/cnb/buildpacks///` -$$ BuildpackPackageLayers(A,\text{linux,arm64}) = ShareLayers(A) + PlatformLayers(A,\text{linux,arm64}) $$ +```bash +├── LICENSE +├── NOTICE +├── README.md +├── buildpack.toml +├── some-file // a file with a big size for example +├── resources +│ └── config.properties +└── bin + ├── build + ├── detect + └── foo +``` -This means the shared layer can be reused in each intermediate image and improve the performance when it contains big files +And the `linux/arm64` version will be similar but with the `bar` file inside the `bin/` folder, as a consequence, the big file +will be duplicated on two different layers increasing the space requirements in the registry ## Builder From 515f41adaf49ab755a074d61c640b02cddc791b9 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 25 Jan 2024 15:07:35 -0500 Subject: [PATCH 163/222] adding some overriding rules Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 86 ++++++++++++++++++++- 1 file changed, 83 insertions(+), 3 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index 54c6d9fc2..995c97bc1 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -284,9 +284,10 @@ versions = [""] ``` - When `more than 1 target is defined` - When `--publish` or `--format file` is specified - - For each `target` an OCI image will be created, following our current process - - `pack` will determine the binaries path (based on the `targets.path` in the buildpack.toml or infering a folder structure similar to the one show above) - - `pack` will include the binaries in the buildpack package OCI image + - For each `target` an OCI image will be created, following these rules + - `pack` will copy everything in the root folder excluding all `{os}/**` folder + - `pack` will determine the **target root binary folder**, this is the specific root folder for a given `target` (based on the `targets.path` in the buildpack.toml or inferring it from a folder structure similar to the one show above) + - `pack` will copy everything from the **target root binary folder** into the base buildpack package folder, in case a conflict file name is found, `pack` will **override** the existing file with the latest one - If more than 1 OCI image was created, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them - When `--daemon` is specified - `pack` can keep using `docker.OSType` to determine the target `os` and probably can do some validations it the `os` is valid target @@ -295,6 +296,85 @@ versions = [""] Let's use some examples to explain the expected behavior in different use cases +#### Overriding files rules + +Let's suppose the Buildpack Author creates a multi-arch folder structure and wants to create multiple buildpack packages + +```bash +. +├── buildpack.toml +├── package.toml +├── foo // some common file for all the platforms +└── linux + ├── amd64 + │ └── bin + │ ├── build + │ ├── detect + │ └── foo + └── arm64 + ├── foo // for some reason we need a different binary for arm64 + └── bin + ├── build + ├── detect + └── bar +``` + +In this case: + - The targets `os` provided are `["linux"]`, then + - For each `target` pack will exclude everything from `linux/**` to be copied into the initial buildpack package + +Resulting into a base structure like: + +```bash +. +└── cnb + └── buildpacks + └── {ID} + └── {version} + ├── buildpack.toml + ├── foo // this is our common file + └── package.toml +``` + +When target is `linux/amd64` + - `pack` will determined **target root binary folder** is `./linux/amd64` + - `pack` will copy everything from the `./linux/arm64` into the base buildpack package folder, because there are no conflicts the expected output will be: + +```bash + +. +└── cnb + └── buildpacks + └── {ID} + └── {version} + ├── bin + │ ├── build + │ ├── detect + │ └── foo // specific platform binary + ├── buildpack.toml + ├── foo // common file + └── package.toml +``` + +On the other hand, When target is `linux/arm64` +- `pack` will determined **target root binary folder** is `./linux/arm64` + - `pack` will copy everything from the `./linux/arm64` into the base buildpack package folder, because there is a conflict trying to copy `./linux/arm64/foo` into `cnb/buildpacks/{ID}/{version}/foo`, `pack` will override it and moves forward, resulting into: + +```bash +. +└── cnb + └── buildpacks + └── {ID} + └── {version} + ├── bin + │ ├── bar // specific platform binary + │ ├── build + │ └── detect + ├── buildpack.toml + ├── foo // override files with the content of the arm64 folder + └── package.toml +``` + #### Buildpacks authors do not use targets AND `platform.os` is not present at `package.toml` This seems to be the case for [Paketo Buildpacks](https://github.com/paketo-buildpacks/java/blob/main/package.toml) From 62847fdb313f022550b8348611421c56b2c0c7ca Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 2 Feb 2024 08:42:33 -0500 Subject: [PATCH 164/222] Update text/0000-multiarch-builders-and-package.md Co-authored-by: Natalie Arellano Signed-off-by: Juan Bustamante Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index 995c97bc1..c6423ddd8 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -396,7 +396,7 @@ let end users know things are changing. ```bash pack buildpack package --config ./package.toml --publish -Warning: A new '--target' flag is available to set the platform for the buildpack package, using 'linux' as default +Warning: A new '--target' flag is available to set the platform for the buildpack package, using 'linux/amd64' as default Successfully published package and saved to registry # Or From 075b11cc4bf44160efb212ac9d8886d2e13ba1af Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 2 Feb 2024 08:42:54 -0500 Subject: [PATCH 165/222] Update text/0000-multiarch-builders-and-package.md Co-authored-by: Natalie Arellano Signed-off-by: Juan Bustamante Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index c6423ddd8..55ad6a25d 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -401,7 +401,7 @@ Successfully published package and saved to registry # Or pack buildpack package --config ./package.toml --format file -Warning: A new '--target' flag is available to set the platform for the buildpack package, using 'linux' as default +Warning: A new '--target' flag is available to set the platform for the buildpack package, using 'linux/amd64' as default Successfully created package and saved to file ``` **Output**: pack will create a buildpack package image (as it is doing it today) with the provided binaries and a From fbdd18122b5eb74d66ac15b9653226509b50f512 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 2 Feb 2024 09:49:13 -0500 Subject: [PATCH 166/222] adding some more feedback from review Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index 55ad6a25d..aa5c8dcf4 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -223,7 +223,7 @@ We propose: - When `platform.os` is present in [package.toml](https://buildpacks.io/docs/reference/config/package-config/), throw a warning messages indicating the field will be removed and `--target` flag must be used - When `platform.os` is not present in [package.toml](https://buildpacks.io/docs/reference/config/package-config/) and `--target` flag is not used, throw a warning messages indicating -a new `--target` flag is available +a new `--target` flag is available and how to use it, or some helpful information on how to add `targets` to the `buildpack.toml` - Keep doing our current process to package a buildpack ### To improve user experience @@ -265,7 +265,7 @@ this will be the way for end-users to specify the platform for which they want t > that the final buildpack image will not change, this will only change the buildpack structure from `pack` perspective In case, this folder structure is not suitable for Buildpack Authors, **we propose** a new `path` attribute to be included -into the `target` section of the `buildpack.toml` to specify where the binaries are located in the filesystem. +into the `target` section of the `buildpack.toml` to specify where the **buildapck root directoy** is located in the filesystem. Based on the [RFC-0096](https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md) the new `buildpack.toml` schema will look like this: From 05a4383d97033a5313fe2a94557bfe33a242592c Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 8 Feb 2024 12:58:40 -0500 Subject: [PATCH 167/222] Removing the shared files complexity Signed-off-by: Juan Bustamante --- text/0000-multiarch-builders-and-package.md | 131 +++++--------------- 1 file changed, 30 insertions(+), 101 deletions(-) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0000-multiarch-builders-and-package.md index aa5c8dcf4..359537d40 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0000-multiarch-builders-and-package.md @@ -32,6 +32,11 @@ that combines them into one single consumable tag for end-users. - Builder: A builder is an image that contains all the components necessary to execute a build. A builder image is created by taking a build image and adding a lifecycle, buildpacks, and files that configure aspects of the build including the buildpack detection order and the location(s) of the run image - Image Manifest: The image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system. See [spec](https://github.com/opencontainers/image-spec/blob/main/manifest.md) - Image Index: The image index is a higher-level manifest which points to specific image manifests, ideal for one or more platforms. See [spec](https://github.com/opencontainers/image-spec/blob/main/image-index.md) +- Buildpack root folder: is the top-most directory where a `buildpack.toml` can be found +- Platform root folder: Based on our new folder structure, the **platform root folder** is the top-most directory that identifies a target in **buildpack root folder** + For example: + - given a target `linux/amd64` the **platform root folder** will be `/linux/amd64` + - given a target `windows/amd64:windows@10.0.20348.1970` the **platform root folder** will be `/windows/amd64/windows@10.0.20348.1970` # Motivation [motivation]: #motivation @@ -239,8 +244,9 @@ this will be the way for end-users to specify the platform for which they want t # Option 1 - no variant is required . ├── buildpack.toml // mandatory +├── package.toml // mandatory └── {os} // optional - └── {arch} // optional + └── {arch} // optional (becomes the platform root folder) └── bin ├── build // platform dependant binary (mandatory) └── detect // platform dependant binary (mandatory) @@ -248,19 +254,24 @@ this will be the way for end-users to specify the platform for which they want t # Option 2 - variant is required . ├── buildpack.toml // mandatory +├── package.toml // mandatory └── {os} // optional └── {arch} // optional └── {variant} // optional - ├── {name@version-1} // optional - │ └── bin - │ ├── build // platform dependant binary (mandatory) - │ └── detect // platform dependant binary (mandatory) - └── {name@version-2} // optional + ├── {name@version-1} // optional (becomes the platform root folder) + │ └── bin + │ ├── build // platform dependant binary (mandatory) + │ └── detect // platform dependant binary (mandatory) + └── {name@version-2} // optional (becomes the platform root folder) └── bin ├── build // platform dependant binary (mandatory) └── detect // platform dependant binary (mandatory) ``` -> **Note** +- `buildpack.toml` file MUST be present at the **buildpack root folder** +- `package.toml` file MUST be present at the **buildpack root folder** +- For each platform, Buildpack Authors are responsible for copying or creating symlink or hard link for files into each **platform root folder** + +> **Note** > For cross-compile buildpacks like Paketo, it looks easy to add a step to their Makefile to compile and separate the binaries following this structure. It is important to mention > that the final buildpack image will not change, this will only change the buildpack structure from `pack` perspective @@ -285,9 +296,8 @@ versions = [""] - When `more than 1 target is defined` - When `--publish` or `--format file` is specified - For each `target` an OCI image will be created, following these rules - - `pack` will copy everything in the root folder excluding all `{os}/**` folder - - `pack` will determine the **target root binary folder**, this is the specific root folder for a given `target` (based on the `targets.path` in the buildpack.toml or inferring it from a folder structure similar to the one show above) - - `pack` will copy everything from the **target root binary folder** into the base buildpack package folder, in case a conflict file name is found, `pack` will **override** the existing file with the latest one + - `pack` will determine the **platform root folder**, this is the specific root folder for a given `target` (based on the `targets.path` in the buildpack.toml or inferring it from a folder structure similar to the one show above) + - `pack` will execute the current process to create a buildpack package using the **platform root folder** and the `target` values - If more than 1 OCI image was created, an [image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) will be created to combine them - When `--daemon` is specified - `pack` can keep using `docker.OSType` to determine the target `os` and probably can do some validations it the `os` is valid target @@ -296,15 +306,14 @@ versions = [""] Let's use some examples to explain the expected behavior in different use cases -#### Overriding files rules +#### How to determine the platform root folder Let's suppose the Buildpack Author creates a multi-arch folder structure and wants to create multiple buildpack packages ```bash . ├── buildpack.toml -├── package.toml -├── foo // some common file for all the platforms +├── package.toml └── linux ├── amd64 │ └── bin @@ -312,33 +321,15 @@ Let's suppose the Buildpack Author creates a multi-arch folder structure and wan │ ├── detect │ └── foo └── arm64 - ├── foo // for some reason we need a different binary for arm64 + ├── foo └── bin ├── build ├── detect └── bar ``` -In this case: - - The targets `os` provided are `["linux"]`, then - - For each `target` pack will exclude everything from `linux/**` to be copied into the initial buildpack package - -Resulting into a base structure like: - -```bash -. -└── cnb - └── buildpacks - └── {ID} - └── {version} - ├── buildpack.toml - ├── foo // this is our common file - └── package.toml -``` - -When target is `linux/amd64` - - `pack` will determined **target root binary folder** is `./linux/amd64` - - `pack` will copy everything from the `./linux/arm64` into the base buildpack package folder, because there are no conflicts the expected output will be: +- When `linux/amd64` the **platform root folder** determined is `/linux/amd64`, and the expected +folder structure in the OCI image for each buildpack package will be: ```bash @@ -352,14 +343,11 @@ When target is `linux/amd64` │ ├── detect │ └── foo // specific platform binary ├── buildpack.toml - ├── foo // common file └── package.toml ``` -On the other hand, When target is `linux/arm64` -- `pack` will determined **target root binary folder** is `./linux/arm64` - - `pack` will copy everything from the `./linux/arm64` into the base buildpack package folder, because there is a conflict trying to copy `./linux/arm64/foo` into `cnb/buildpacks/{ID}/{version}/foo`, `pack` will override it and moves forward, resulting into: - +On the other hand, When target is `linux/arm64, the **platform root folder** determined is `/linux/arm64` +and the output OCI image folder structure looks like: ```bash . └── cnb @@ -371,7 +359,7 @@ On the other hand, When target is `linux/arm64` │ ├── build │ └── detect ├── buildpack.toml - ├── foo // override files with the content of the arm64 folder + ├── foo └── package.toml ``` @@ -759,63 +747,6 @@ with a content similar to: If the Buildpack Author wants to create a single buildpack package they will use the `target` flag, similar to our previous examples. -### Important considerations for sharing files with the new folder structure - -Let's suppose we have a multi-arch buildpack **A** with local folder structure like this: - -```bash -. -├── LICENSE -├── NOTICE -├── README.md -├── buildpack.toml -├── package.toml -├── some-file // a file with a big size for example -├── resources -│ ├── config.properties -└── linux - ├── amd64 - │ └── bin - │ ├── build - │ ├── detect - │ └── foo - └── arm64 - └── bin - ├── build - ├── detect - └── bar -``` - -As we already discussed in this RFC, we are expecting to create two OCI images artifacts and combine with an -[image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). The current [distribution spec](https://github.com/buildpacks/spec/blob/main/distribution.md#buildpackage) -defines: - -``` -Each buildpack layer blob MUST contain a single buildpack at the following file path: - -/cnb/buildpacks/// -``` - -This means, we will end up duplicating the big file in each OCI images, for example our `linux/amd64` buildpack package -will have 1 layer with the following content under `/cnb/buildpacks///` - -```bash -├── LICENSE -├── NOTICE -├── README.md -├── buildpack.toml -├── some-file // a file with a big size for example -├── resources -│ └── config.properties -└── bin - ├── build - ├── detect - └── foo -``` - -And the `linux/arm64` version will be similar but with the `bar` file inside the `bin/` folder, as a consequence, the big file -will be duplicated on two different layers increasing the space requirements in the registry - ## Builder Similar to how we did it for the `buildpack package`, lets summaries, our current process to create a **Builder**: @@ -1005,10 +936,6 @@ Tip: Run pack build --builder to use this builder **Output**: We keep our current behavior and detect the `os` and `architecture` from the daemon. Because there is `target` that matches the daemon `os/arch` the builder is being built. -Using `--target` flag against the daemon with a different platform - -TODO check if this is possible! - @@ -12,7 +12,7 @@ # Summary [summary]: #summary -The problem for adding support for multi-arch buildpacks can be divided into three parts: +The problem for adding support for multi-platform buildpacks can be divided into three parts: 1. Support buildpack authors to **migrate their existing buildpacks** to support multiple operating systems, architectures, variants and distros. 2. Support buildpack authors to **create new buildpacks and builders** that handle multi-arch from the beginning. 3. Support application developers to **create application images** using multi-arch buildpacks and builders. @@ -240,7 +240,7 @@ this will be the way for end-users to specify the platform for which they want t - Add `targets` section to `buildpack.toml` to help Buildpack Authors to include support for new platforms without having to update their `pack buildpack package` command in their CI/CD pipelines -- A new folder structure to organize the buildpacks binaries for multi-arch images, similar to this one: +- A new folder structure to organize the buildpacks binaries for multi-platform images, similar to this one: ```bash # Option 1 - no variant is required . @@ -310,7 +310,7 @@ Let's use some examples to explain the expected behavior in different use cases #### How to determine the platform root folder -Let's suppose the Buildpack Author creates a multi-arch folder structure and wants to create multiple buildpack packages +Let's suppose the Buildpack Author creates a multi-platform folder structure and wants to create multiple buildpack packages ```bash . @@ -432,7 +432,7 @@ what about creating a multi-platform image for several targets? ```bash pack buildpack package --config ./package.toml --publish --target linux/arm64 --target linux/amd64 -A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64' +A multi-platform buildpack package will be created for: 'linux/amd64', 'linux/arm64' Successfully published package and saved to registry # Or @@ -693,7 +693,7 @@ id = "*" When Buildpack Authors want to create a multi-arch images, they can execute the following command: ```bash pack buildpack package --config ./package.toml --publish -Info: A multi-arch buildpack package will be created for target platforms: 'linux/amd64', 'linux/arm64', 'windows/amd64' +Info: A multi-platform buildpack package will be created for targets: 'linux/amd64', 'linux/arm64', 'windows/amd64' Successfully published package and saved to registry ``` A fully multi-arch buildpack will be created automatically, because we have more than one target defined @@ -1002,7 +1002,7 @@ What about multi-architecture builders? ```bash pack builder create --config ./builder.toml --publish -Info: A multi-arch builder will be created for targets platform: 'linux/amd64', 'linux/arm64' +Info: A multi-platform builder will be created for targets: 'linux/amd64', 'linux/arm64' Successfully created builder image Tip: Run pack build --builder to use this builder ``` @@ -1014,7 +1014,7 @@ pack builder create --config ./builder.toml \ --target linux/amd64 \ --target linux/arm64 \ --publish -Info: A multi-arch builder will be created for targets platform: 'linux/amd64', 'linux/arm64' +Info: A multi-platform builder will be created for targets: 'linux/amd64', 'linux/arm64' Successfully created builder image Tip: Run pack build --builder to use this builder ``` From 822d70252536a5fe87db879bd38e87fde169c261 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Wed, 10 Apr 2024 10:42:56 -0400 Subject: [PATCH 173/222] RFC 0128 [#295] Signed-off-by: Natalie Arellano --- ...nd-package.md => 0128-multiarch-builders-and-package.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-multiarch-builders-and-package.md => 0128-multiarch-builders-and-package.md} (99%) diff --git a/text/0000-multiarch-builders-and-package.md b/text/0128-multiarch-builders-and-package.md similarity index 99% rename from text/0000-multiarch-builders-and-package.md rename to text/0128-multiarch-builders-and-package.md index d43f1ae7d..3d3844985 100644 --- a/text/0000-multiarch-builders-and-package.md +++ b/text/0128-multiarch-builders-and-package.md @@ -3,10 +3,10 @@ - Name: Multi-platform support for builders and buildpack packages - Start Date: 2023-09-14 - Author(s): @jjbustamante -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#295](https://github.com/buildpacks/rfcs/pull/295) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: N/A - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Summary From 8c2a6a0bdc1b49d5e3d537bc6508e30fb5276c85 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 27 Mar 2024 23:43:12 +0000 Subject: [PATCH 174/222] Update RFC96 to match what landed in the spec A few changes were made to what was in the RFC when it came to update the spec. This backports those changes to the RFC to try and prevent user confusion if they use the RFC as a means of learning about targets when implementing. Specifically: - The `io.buildpacks.distribution.*` Docker labels were renamed to `io.buildpacks.base.distro.*`. - The `io.buildpacks.id` Docker label was renamed to `io.buildpacks.base.id`. - The `buildpack.toml` table `[[targets.distributions]]` was renamed to `[[targets.distros]]` and the `versions` field within it renamed to `version` (along with its type changing from an array to a string). See: https://github.com/buildpacks/spec/blob/platform/0.13/platform.md#target-data Signed-off-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0096-remove-stacks-mixins.md | 44 ++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 12 deletions(-) diff --git a/text/0096-remove-stacks-mixins.md b/text/0096-remove-stacks-mixins.md index 34f17ea98..10a584e5d 100644 --- a/text/0096-remove-stacks-mixins.md +++ b/text/0096-remove-stacks-mixins.md @@ -41,11 +41,11 @@ Instead of a stack ID, runtime and build-time base images must contain the follo - OS (e.g., "linux", `$GOOS`), specified as `os` in the base image `config` - Architecture (e.g., "arm", `$GOARCH`), specified as `architecture` in the base image `config` - Architecture Variant (optional) (e.g., "v6", `$GOARM`), specified as `variant` in the base image `config` -- Distribution (optional) (e.g., "ubuntu", `$ID`), specified as a label `io.buildpacks.distribution.name` -- Version (optional) (e.g., "18.04", `$VERSION_ID`), specified as a label `io.buildpacks.distribution.version` +- Distribution (optional) (e.g., "ubuntu", `$ID`), specified as a label `io.buildpacks.base.distro.name` +- Version (optional) (e.g., "18.04", `$VERSION_ID`), specified as a label `io.buildpacks.base.distro.version` Additionally, the runtime base may contain the following metadata: -- Target ID (optional) (e.g., "minimal"), specified as a label `io.buildpacks.id` +- Target ID (optional) (e.g., "minimal"), specified as a label `io.buildpacks.base.id` OS, Architecture, and Architecture Variant must be valid identifiers as defined in the [OCI Image specification](https://github.com/opencontainers/image-spec/blob/main/config.md). @@ -53,7 +53,7 @@ Target ID is an identifier specified on the runtime base image that must be prov This allows buildpacks to change their behavior if a run image is selected (e.g., distroless) that has special properties outside of OS, architecture, etc. For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). -The `os.version` field in an base image `config` may contain combined distribution and version information, but it is not used by the lifecycle. +The `os.version` field in a base image `config` may contain combined distribution and version information, but it is not used by the lifecycle. For Windows-based images, Distribution should be empty. Version should be the [suggested value of `os.version`](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) in the OCI spec (e.g., `10.0.14393.1066`). @@ -81,17 +81,17 @@ versions = ["18.04", "20.04"] [[targets]] os = "linux" arch = "amd64" -[[targets.distributions]] +[[targets.distros]] name = "ubuntu" -versions = ["14.04", "16.04"] +version = "16.04" [[targets]] os = "linux" arch = "arm" variant = "v6" -[[targets.distributions]] +[[targets.distros]] name = "ubuntu" -versions = ["14.04", "16.04"] +version = "16.04" ``` ## Runtime Metadata @@ -153,11 +153,11 @@ If the newly-specified field values are missing, the lifecycle and pack may used ``` config.os = "linux" config.architecture = "amd64" -io.buildpacks.distribution.name = "ubuntu" -io.buildpacks.distribution.version = "18.04" +io.buildpacks.base.distro.name = "ubuntu" +io.buildpacks.base.distro.version = "18.04" ``` -Moving forward it's encouraged for buildpack authors to support both `[[stacks]]` and `[[targets]]` sections in `buildpack.toml` for maximum compatibility. In order to ease this process for those using the `io.buildpacks.stacks.bionic`, lifecycle will translate any section that sets this as on of the `stacks`: +Moving forward it's encouraged for buildpack authors to support both `[[stacks]]` and `[[targets]]` sections in `buildpack.toml` for maximum compatibility. In order to ease this process for those using the `io.buildpacks.stacks.bionic`, lifecycle will translate any section that sets this as one of the `stacks`: ```toml [[stacks]] @@ -170,7 +170,7 @@ to [[targets]] os = "linux" arch = "amd64" -[[targets.distributions]] +[[targets.distros]] name = "ubuntu" versions = ["18.04"] ``` @@ -210,3 +210,23 @@ rename x86_64 -> amd64 in keeping with all other usages of arch. descriptors. ### Motivation This is how we do it everywhere else, this is the way. + +## Amended +### Meta +[meta-1]: #meta-1 +- Name: Rename Docker labels and `buildpack.toml` table names +- Start Date: 2024-04-08 +- Author(s): @edmorley +- Amendment Pull Request: [rfcs#310](https://github.com/buildpacks/rfcs/pull/310) + +### Summary + +Changes were made to the Docker label and `buildpack.toml` table names between when this RFC was written and the changes were made to the spec in [spec#365](https://github.com/buildpacks/spec/pull/365), which have now been backported to the RFC: + +- The `io.buildpacks.distributions.*` Docker labels were renamed to `io.buildpacks.base.distro.*`. +- The `io.buildpacks.id` Docker label was renamed to `io.buildpacks.base.id`. +- The `buildpack.toml` table `[[targets.distributions]]` was renamed to `[[targets.distros]]` and the `versions` field within it renamed to `version` (along with its type changing from an array to a string). + +### Motivation + +To prevent use of the wrong Docker label or `buildpack.toml` table names, if users base their implementations on the RFC rather than reading the spec. From ff8c235b14be3afa3c3a3215d882b1428d47fbcf Mon Sep 17 00:00:00 2001 From: Aidan Delaney Date: Thu, 11 Apr 2024 09:34:02 +0100 Subject: [PATCH 175/222] Remove Windows Containers support Remove support for Windows Containers in CNB tooling and documentation, but retain support in the Buildpacks spec. Signed-off-by: Aidan Delaney --- .../0000-remove-windows-containers-support.md | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 text/0000-remove-windows-containers-support.md diff --git a/text/0000-remove-windows-containers-support.md b/text/0000-remove-windows-containers-support.md new file mode 100644 index 000000000..eb988ecf4 --- /dev/null +++ b/text/0000-remove-windows-containers-support.md @@ -0,0 +1,97 @@ +# Meta +[meta]: #meta +- Name: Remove Windows Containers Support +- Start Date: 2024-04-11 +- Author(s): aidan.delaney +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: "N/A" + +# Summary +[summary]: #summary + +Retain Windows Containers support in the buildpacks specification, but remove Windows Containers support from `lifecycle`, `pack` CLI, and documentation. + +# Definitions +[definitions]: #definitions + +Make a list of the definitions that may be useful for those reviewing. Include phrases and words that buildpack authors or other interested parties may not be familiar with. + +* [Windows Containers](https://learn.microsoft.com/en-us/virtualization/windowscontainers/about/): Containers derived from Windows Server Containers. For example: `mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2022` + +# Motivation +[motivation]: #motivation + +- There are no known users of Windows Container support within the buildpacks ecosystem. +- Supporting Windows Containers is straightforward at a specification level. However, the CNB project has neither the time nor the expertise to continue to support Windows Containers in our tooling. +- Removing Windows Container support from CNB tooling will speed up development, simplify testing and impact no existing users. +- Establish platform parity between `pack` and `kpack`. `kpack` has never supported building Windows Containers. + +# What it is +[what-it-is]: #what-it-is + +- We want to remove Windows Containers related documentation, but clearly state that the specification is platform neutral. +- Remove Windows Containers support from `pack`, but retain the Windows platform release of the `pack` CLI. + * this ensures `pack` users can still invoke `pack` on a Command or Power Shell console +- Remove Windows Containers support from `lifecycle`. Lifecycle can remove the Windows platform release as it will then be guaranteed to run on a Linux-based container. + +# How it Works +[how-it-works]: #how-it-works + + +# Migration +[migration]: #migration + +Teams that require Windows Container support will have to + +* continue to use the last release of `pack` that supports Windows Containers, +* fork `pack` and `lifecycle` to maintain Windows Containers support, or +* adopt alternative tooling that supports Windows Containers. + +# Drawbacks +[drawbacks]: #drawbacks + +Windows Containers support is well tested within `pack`. Removing support for Windows Containers removes a unique selling point of the `pack` CLI and `lifecycle` + +# Alternatives +[alternatives]: #alternatives + +- We have discussed platform support with `pack` users. There are no `pack` users who want to maintain Windows Container support. Should engineering effort become available to help with development and maintenance, then this proposal can be dropped. + +# Prior Art +[prior-art]: #prior-art + +N/A + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- We have made outreach efforts to CNB users to survey requirements of Windows Container support. We have found no users. The number of CNB users that require Windows Container support is still an open question. + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +This proposal requires no specification changes. + +# History +[history]: #history + + From 55cfde33a629f126b7de3b07b4e84fea383f0d81 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Thu, 11 Apr 2024 17:39:49 +0100 Subject: [PATCH 176/222] RFC 0128: Use correct `[[targets.distros]]` schema Updates RFC 0128 to use the correct `[[targets.distros]]` schema as defined already in the spec: https://github.com/buildpacks/spec/blob/main/buildpack.md#buildpacktoml-toml I've not added an `## Amended` section, since: 1. The RFC has only just merged (this is effectively a review comment that missed the merge by a day) 2. The parts of the RFC being fixed are existing concepts already in the spec, rather than design decisions relating to the purpose of the RFC itself. Fixes: https://github.com/buildpacks/rfcs/pull/295#discussion_r1559614121 Signed-off-by: Ed Morley <501702+edmorley@users.noreply.github.com> --- text/0128-multiarch-builders-and-package.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/text/0128-multiarch-builders-and-package.md b/text/0128-multiarch-builders-and-package.md index 3d3844985..e36d1d2bc 100644 --- a/text/0128-multiarch-builders-and-package.md +++ b/text/0128-multiarch-builders-and-package.md @@ -95,9 +95,9 @@ arch = "arm64" os = "windows" arch = "amd64" -[[targets.distributions]] +[[targets.distros]] name = "windows" -versions = ["10.0.20348.1970"] +version = "10.0.20348.1970" # Stacks (deprecated) the buildpack will work with [[stacks]] @@ -286,9 +286,9 @@ variant = "" # optional path = "" -[[targets.distributions]] +[[targets.distros]] name = "" -versions = [""] +version = "" ``` - When `more than 1 target is defined` - When `--publish` is specified @@ -681,9 +681,9 @@ arch = "arm64" os = "windows" arch = "amd64" -[[targets.distributions]] +[[targets.distros]] name = "windows" -versions = ["10.0.20348.1970"] +version = "10.0.20348.1970" # Stacks (deprecated) the buildpack will work with [[stacks]] @@ -763,9 +763,9 @@ os = "" arch = "" variant = "" -[[targets.distributions]] +[[targets.distros]] name = "" -versions = [""] +version = "" [[dependencies]] uri = "" @@ -843,9 +843,9 @@ image = "" os = "" arch = "" variant = "" -[[targets.distributions]] +[[targets.distros]] name = "" -versions = [""] +version = "" ``` - Add a new `--target` optional flag with format `[os][/arch][/variant]:[name@version]` to create a builder for a particular target, this will help end-users to specify the platform for which they want to create single OCI artifact. From 6c56aa24356633773b000f72e9a7b074306eda32 Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 11 Apr 2024 14:52:28 -0500 Subject: [PATCH 177/222] export run image metadata Signed-off-by: Joey Brown --- text/0000-export-run-image-metadata.md | 121 +++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 text/0000-export-run-image-metadata.md diff --git a/text/0000-export-run-image-metadata.md b/text/0000-export-run-image-metadata.md new file mode 100644 index 000000000..6ef3802fa --- /dev/null +++ b/text/0000-export-run-image-metadata.md @@ -0,0 +1,121 @@ +# Meta +[meta]: #meta +- Name: Export Run Image Metadata +- Start Date: 2024-04-11 +- Author(s): joeybrown-sf +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: N/A + +# Summary +[summary]: #summary + +The `report.toml` file created by the lifecycle exporter and rebaser should include `run-image.image`, `run-image.reference`, and `run-image.top-layer`. These values can be critical to a platform rebase process. + +# Motivation +[motivation]: #motivation + +When publishing an image directly to a registry, parsing the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json) is inefficient--it requires a call to the registry after the image is published. Adding these values to `report.toml` lets platforms efficiently access this data after an export or rebase without a call to the registry. + +# What it is +[what-it-is]: #what-it-is + +These values will be included when the lifecycle exporter/creator/rebaser binary writes `report.toml`. + +Here are two examples of `report.toml` content. (Other values are omitted for readability.) + +#### Image published to a registry: +``` +[image] +tags = ... +digest = ... +image-id = ... +manifest-size = ... + +[run-image] +image = "run/name:foo" +reference = "index.docker.io/run/name@sha256:94f85561b0976bf1e2bef6b14de92299ebcd4c8148802cf9b217654651e4f416" +top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +``` + +#### Image exported to the docker daemon: +``` +[image] +tags = ... +digest = ... +image-id = ... +manifest-size = ... + +[run-image] +image = "run/name:foo" +reference = "5b90f9c0e189" +top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +``` + +# How it Works +[how-it-works]: #how-it-works + +This metadata is readily available when `report.toml` is created, so it will be straight-forward to extend `report.toml`. + +# Migration +[migration]: #migration + +N/A + +This is an additive change to a metadata file and will be backwards compatible. + +# Drawbacks +[drawbacks]: #drawbacks + +This metadata is written to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json). It can be accessed by querying the config from docker daemon or registry. So we will be writing this data to two outputs. + +# Alternatives +[alternatives]: #alternatives + +Do nothing and continue to require platforms to retrieve this information via alternative means--either querying the docker daemon or registry. + +Introduce a new output file (`lifecycle-metadata.toml?`) that contains everything in [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json). + +# Prior Art +[prior-art]: #prior-art + +These values are among those that `pack inspect` returns. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes +[spec-changes]: #spec-changes + +The following would be appended to the [`report.toml` spec](https://github.com/buildpacks/spec/blob/main/platform.md#reporttoml-toml) (this section would be materially identical to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json)): + +> run-image.top-layer MUST contain the uncompressed digest of the top layer of the run-image. +> +>run-image.reference MUST uniquely identify the run image. It MAY contain one of the following +> - An image ID (the digest of the uncompressed config blob) +> - A digest reference to a manifest stored in an OCI registry + +# History +[history]: #history + + From 0af349b834dc7396052fd97d1ed2d91894033b1f Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 18 Apr 2024 09:01:29 -0500 Subject: [PATCH 178/222] provide more motivation content & alternatives Signed-off-by: Joey Brown --- text/0000-export-run-image-metadata.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/text/0000-export-run-image-metadata.md b/text/0000-export-run-image-metadata.md index 6ef3802fa..8d9922674 100644 --- a/text/0000-export-run-image-metadata.md +++ b/text/0000-export-run-image-metadata.md @@ -12,12 +12,16 @@ # Summary [summary]: #summary -The `report.toml` file created by the lifecycle exporter and rebaser should include `run-image.image`, `run-image.reference`, and `run-image.top-layer`. These values can be critical to a platform rebase process. +The `report.toml` file created by the lifecycle exporter and rebaser should include `run-image.image`, `run-image.reference`, and `run-image.top-layer`. These values are not necessarily known prior to export or rebase they can be critical to a platform rebase process. # Motivation [motivation]: #motivation -When publishing an image directly to a registry, parsing the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json) is inefficient--it requires a call to the registry after the image is published. Adding these values to `report.toml` lets platforms efficiently access this data after an export or rebase without a call to the registry. +Platform operators may need a comprehensive understanding of images on their platform in order to make decisions about rebase and image publishing. Run image metadata is likely part of this comprehensive understanding for rebase. It is likely that this data may only be known after an image is created or rebased, and today it is only accessible via reading the image. Therefore, in order to access this metadata, platform operators must query the image. + +Querying the docker daemon or querying an image registry is suboptimal and we should make this data more accessible. It is suboptimal because it requires the platform to run an additional service to query the data it just published. If we make this data more accessible, we could potentially reduce image queries (registry calls) calls by a significant factor. + +Putting this data into `report.toml` is advantageous over other methods, especially when considering the kubernetes `terminationMessagePath` message [pattern](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#pod-v1-core). In this pattern, the content of `report.toml` can be used as a container's termination message, making this data easily accessible after an image is exported or rebased within a kubernetes container. # What it is [what-it-is]: #what-it-is @@ -69,19 +73,27 @@ This is an additive change to a metadata file and will be backwards compatible. # Drawbacks [drawbacks]: #drawbacks -This metadata is written to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json). It can be accessed by querying the config from docker daemon or registry. So we will be writing this data to two outputs. +This metadata is written to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json) and it can be accessed by querying a docker daemon or registry. So we will be writing this data to two outputs. # Alternatives [alternatives]: #alternatives -Do nothing and continue to require platforms to retrieve this information via alternative means--either querying the docker daemon or registry. +1. Do nothing and continue to require platforms to retrieve this information via alternative means--either querying the docker daemon or registry. + - Rebase process may remain suboptimal for some platform providers. + +2. Write all the metadata labels to `report.toml`. + - This could break platform operators that are using the `terminationMessagePath` [pattern](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#pod-v1-core). Because some of the metadata includes unbounded arrays, we could explode this report beyond the max size of 4096 bytes. -Introduce a new output file (`lifecycle-metadata.toml?`) that contains everything in [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json). +3. Write another file that contains this metadata (and potentially more metadata). + - If we consider this approach and take it to the logical conclusion, we should consider writing a sparse image as output. A sparse image would contain all the metadata available, and it would be in a well-known format that other image tools like `crane`, `skopeo`, or `pack` can read. + - Writing to another file is not as simple as writing to `report.toml`. + - It increases the complexity of lifecycle exporter & rebaser. + - Writing to another file would not give platform operators the advantage of the `terminationMessagePath` [pattern](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#pod-v1-core), because they are likely already reading `report.toml`. # Prior Art [prior-art]: #prior-art -These values are among those that `pack inspect` returns. +These values are written to image labels. And they are among values that `pack inspect` returns. # Unresolved Questions [unresolved-questions]: #unresolved-questions From 7ccbed177ec86c33b54c939fcbef543e9662d280 Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 18 Apr 2024 11:27:06 -0500 Subject: [PATCH 179/222] update to add mirrors and change casing Signed-off-by: Joey Brown --- text/0000-export-run-image-metadata.md | 32 +++++++++++++++----------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/text/0000-export-run-image-metadata.md b/text/0000-export-run-image-metadata.md index 8d9922674..394a0c956 100644 --- a/text/0000-export-run-image-metadata.md +++ b/text/0000-export-run-image-metadata.md @@ -12,7 +12,13 @@ # Summary [summary]: #summary -The `report.toml` file created by the lifecycle exporter and rebaser should include `run-image.image`, `run-image.reference`, and `run-image.top-layer`. These values are not necessarily known prior to export or rebase they can be critical to a platform rebase process. +The `report.toml` file created by the lifecycle exporter and rebaser should include the following properties: +- `run-image.image` +- `run-image.mirrors` +- `run-image.reference` +- `run-image.top-layer` + +These values are not necessarily known prior to export or rebase they can be critical to a platform rebase process. # Motivation [motivation]: #motivation @@ -37,11 +43,10 @@ tags = ... digest = ... image-id = ... manifest-size = ... - -[run-image] -image = "run/name:foo" -reference = "index.docker.io/run/name@sha256:94f85561b0976bf1e2bef6b14de92299ebcd4c8148802cf9b217654651e4f416" -top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +run-image.image = "run/name:foo" +run-image.reference = "index.docker.io/run/name@sha256:94f85561b0976bf1e2bef6b14de92299ebcd4c8148802cf9b217654651e4f416" +run-image.top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +run-image.mirrors = ["", ""] ``` #### Image exported to the docker daemon: @@ -51,11 +56,10 @@ tags = ... digest = ... image-id = ... manifest-size = ... - -[run-image] -image = "run/name:foo" -reference = "5b90f9c0e189" -top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +run-image.image = "run/name:foo" +run-image.reference = "5b90f9c0e189" +run-image.top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +run-image.mirrors = ["", ""] ``` # How it Works @@ -105,11 +109,13 @@ N/A The following would be appended to the [`report.toml` spec](https://github.com/buildpacks/spec/blob/main/platform.md#reporttoml-toml) (this section would be materially identical to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json)): -> run-image.top-layer MUST contain the uncompressed digest of the top layer of the run-image. +> runImage.topLayer MUST contain the uncompressed digest of the top layer of the run-image. > ->run-image.reference MUST uniquely identify the run image. It MAY contain one of the following +> runImage.reference MUST uniquely identify the run image. It MAY contain one of the following > - An image ID (the digest of the uncompressed config blob) > - A digest reference to a manifest stored in an OCI registry +> +> runImage.image and runImage.mirrors MUST be resolved from run.toml from the given # History [history]: #history From fcbe0f3deb4f0d62ddf9381a3043de8119e6975a Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 18 Apr 2024 12:18:12 -0500 Subject: [PATCH 180/222] add a reference to the RFC that introduces report.toml Signed-off-by: Joey Brown --- text/0000-export-run-image-metadata.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-export-run-image-metadata.md b/text/0000-export-run-image-metadata.md index 394a0c956..6bd8013f7 100644 --- a/text/0000-export-run-image-metadata.md +++ b/text/0000-export-run-image-metadata.md @@ -99,6 +99,8 @@ This metadata is written to the [`lifecycle.metadata` label](https://github.com/ These values are written to image labels. And they are among values that `pack inspect` returns. +`report.toml` was introduced in [RFC 0040](/text/0040-export-report.md). That RFC includes some context for why `report.toml` exists. + # Unresolved Questions [unresolved-questions]: #unresolved-questions From 7dc794a22741840df53b28b3062b3f86d76419ba Mon Sep 17 00:00:00 2001 From: Craig Andrews Date: Wed, 26 Jun 2024 16:06:14 -0400 Subject: [PATCH 181/222] RFC for OCI Image Annotations on Buildpacks Signed-off-by: Craig Andrews --- text/0000-oci-image-annotations.md | 116 +++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 text/0000-oci-image-annotations.md diff --git a/text/0000-oci-image-annotations.md b/text/0000-oci-image-annotations.md new file mode 100644 index 000000000..00ca7089c --- /dev/null +++ b/text/0000-oci-image-annotations.md @@ -0,0 +1,116 @@ +# Meta +[meta]: #meta +- Name: OCI Image Annotations on Buildpacks +- Start Date: 2024-06-26 +- Author(s): @candrews +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +The `buildpack` tool should set OCI annotations on the OCI artifacts it produces providing users of these buildpacks with a consistent, standard mechanism for gathering information about the buildpack, including how to find its documentation and what version it is. The annotation values can be gathered from existing data sources (such as `buildpack.toml`) therefore not requiring any additional effort on the part of users of the `buildpack` tool. + +# Definitions +[definitions]: #definitions + +[Buildpacks](https://buildpacks.io/docs/for-app-developers/concepts/buildpack/) are [OCI images](https://github.com/opencontainers/image-spec/blob/v1.1.0/README.md). [Annotations](https://github.com/opencontainers/image-spec/blob/v1.1.0/annotations.md) are optional properties that can be applies to image manifests and descriptions providing mechanism to communicate metadata. The [Pre-Defined Annotation Keys](https://github.com/opencontainers/image-spec/blob/v1.1.0/annotations.md#pre-defined-annotation-keys) are a standardized set of annotations that can be used to convey metadata in a consistent way between image authors and users. + +# Motivation +[motivation]: #motivation + +Knowing the origin and other metadata for a buildpack (which is an OCI image) is very helpful. Some examples of such use cases include finding release notes, user manuals, bug reporting procedures, and license information. Currently, it can be difficult to find the source control repository of a buildpack as that information is not available in a standard way. + +The OCI Image Format Specification's Pre-Defined Annotation Keys provide a standardized way to discover additional information about an OCI image. Because these annotations are standardized and widely used, tools have come to use them. For example, [Snyk](https://snyk.io/blog/how-and-when-to-use-docker-labels-oci-container-annotations/) and [Renovate](https://github.com/renovatebot/renovate/blob/34.115.1/lib/modules/datasource/docker/readme.md) use these annotations. + +The outcome will be that users and tools will be able to gather more information about buildpacks, facilitating use cases such as gathering releases notes and finding documentation. + +# What it is +[what-it-is]: #what-it-is + +`pack buildpack package` should set the following OCI annotations on the images it produces: + +- `org.opencontainers.image.source` +- `org.opencontainers.image.revision` +- `org.opencontainers.image.title` +- `org.opencontainers.image.version` +- `org.opencontainers.image.url` (when possible) +- `org.opencontainers.image.description` (when possible) + +The target personas as buildpack users, platform operators, and platform implementers. Any of those groups will be able to more easily understand the origin (source), version, and other information about the buildpack. This information can then be used manually or with the aid of tools to get release notes which aid these personas in making informed decisions. + +# How it Works +[how-it-works]: #how-it-works + +When building the buildpack, the `buildpack` tool can get the values for the `org.opencontainers.image.source` and `org.opencontainers.image.revision` annotations from git. `org.opencontainers.image.source` is derived from the git origin and `org.opencontainers.image.revision` is the git commit ref. + +The other annotation values come from `buildpack.toml` mapped to OCI annotations as follows: + +- `name` -> `org.opencontainers.image.title` +- `version` -> `org.opencontainers.image.version` +- `homepage` (optional) -> `org.opencontainers.image.url` +- `description` (optional) -> `org.opencontainers.image.description` + +The following example values are from [Paketo Buildpack for Java 13.0.1](https://github.com/paketo-buildpacks/java/releases/tag/v13.0.1): + +- `org.opencontainers.image.source`: https://github.com/paketo-buildpacks/java +- `org.opencontainers.image.revision`: 09747b1df0a56aea74ce9b01af89df6feb1fc50a +- `org.opencontainers.image.title`: Paketo Buildpack for Java +- `org.opencontainers.image.version`: 13.0.1 +- `org.opencontainers.image.url`: https://paketo.io/docs/howto/java +- `org.opencontainers.image.description`: A Cloud Native Buildpack with an order definition suitable for Java applications + +# Migration +[migration]: #migration + +The `buildpack` tool would be modified to set the annotations. + +# Drawbacks +[drawbacks]: #drawbacks + +N/A + +# Alternatives +[alternatives]: #alternatives + +Instead of standardizing the use of these annotations across all buildpacks, each buildpack could add the annotations individually. However, that approach has significant consistency and maintainability concerns. Standardizing the annotations and implementing them consistently across all buildpacks minimizes risk and maximizes utility. If this approach is not done, users will continue to be unable to use tools to gather buildpack information, and gathering that information manually will continue to be difficult or impossible. + +# Prior Art +[prior-art]: #prior-art + +Many images are setting OCI image annotations with adoption continually on the rise. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +The [distribution spec](https://github.com/buildpacks/spec/blob/main/distribution.md) would be updated to document the OCI image annotations as covered in ["How it Works"](#how-it-works). + + +# History +[history]: #history + + \ No newline at end of file From fb53ceb92f323010516d3e9e932b185326092522 Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Mon, 8 Jul 2024 14:55:23 -0500 Subject: [PATCH 182/222] Update and rename 0000-export-run-image-metadata.md to 0129-export-run-image-metadata.md Updating for merge Signed-off-by: Jesse Brown Signed-off-by: Jesse Brown --- ...-image-metadata.md => 0129-export-run-image-metadata.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-export-run-image-metadata.md => 0129-export-run-image-metadata.md} (96%) diff --git a/text/0000-export-run-image-metadata.md b/text/0129-export-run-image-metadata.md similarity index 96% rename from text/0000-export-run-image-metadata.md rename to text/0129-export-run-image-metadata.md index 6bd8013f7..124b24088 100644 --- a/text/0000-export-run-image-metadata.md +++ b/text/0129-export-run-image-metadata.md @@ -3,10 +3,10 @@ - Name: Export Run Image Metadata - Start Date: 2024-04-11 - Author(s): joeybrown-sf -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: https://github.com/buildpacks/rfcs/pull/313 - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: https://github.com/buildpacks/lifecycle/issues/1372 - Supersedes: N/A # Summary From 04e957ea7054f5f4b404040085b5cf9a3fdc0710 Mon Sep 17 00:00:00 2001 From: Craig Andrews Date: Thu, 11 Jul 2024 15:48:05 -0400 Subject: [PATCH 183/222] Note that buildpacks without these annotation would still be valid Co-authored-by: Joe Kutner Signed-off-by: Craig Andrews --- text/0000-oci-image-annotations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-oci-image-annotations.md b/text/0000-oci-image-annotations.md index 00ca7089c..d0d39efb1 100644 --- a/text/0000-oci-image-annotations.md +++ b/text/0000-oci-image-annotations.md @@ -66,7 +66,7 @@ The following example values are from [Paketo Buildpack for Java 13.0.1](https:/ # Migration [migration]: #migration -The `buildpack` tool would be modified to set the annotations. +The `buildpack` tool would be modified to set the annotations. Because the tool _should_ set these annotations (not _must_ set), buildpacks created with earlier versions of the tool are still consider to be valid in accordance with the distribution specification. # Drawbacks [drawbacks]: #drawbacks From dca1e5e4bd2c90ab5bcd68a892cfaa8cf5fc4a8c Mon Sep 17 00:00:00 2001 From: Craig Andrews Date: Fri, 12 Jul 2024 11:33:52 -0400 Subject: [PATCH 184/222] Don't require VCS info, as that info isn't always available Co-authored-by: Joe Kutner Signed-off-by: Craig Andrews --- text/0000-oci-image-annotations.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-oci-image-annotations.md b/text/0000-oci-image-annotations.md index d0d39efb1..ae8edc642 100644 --- a/text/0000-oci-image-annotations.md +++ b/text/0000-oci-image-annotations.md @@ -33,8 +33,8 @@ The outcome will be that users and tools will be able to gather more information `pack buildpack package` should set the following OCI annotations on the images it produces: -- `org.opencontainers.image.source` -- `org.opencontainers.image.revision` +- `org.opencontainers.image.source` (when possible) +- `org.opencontainers.image.revision` (when possible) - `org.opencontainers.image.title` - `org.opencontainers.image.version` - `org.opencontainers.image.url` (when possible) From 1ac82955b7c8265503f70609d7e248e67c79dc9f Mon Sep 17 00:00:00 2001 From: Craig Andrews Date: Fri, 12 Jul 2024 11:37:08 -0400 Subject: [PATCH 185/222] `org.opencontainers.image.revision` should be the hash, not the ref Signed-off-by: Craig Andrews --- text/0000-oci-image-annotations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-oci-image-annotations.md b/text/0000-oci-image-annotations.md index ae8edc642..a77b0a3ea 100644 --- a/text/0000-oci-image-annotations.md +++ b/text/0000-oci-image-annotations.md @@ -45,7 +45,7 @@ The target personas as buildpack users, platform operators, and platform impleme # How it Works [how-it-works]: #how-it-works -When building the buildpack, the `buildpack` tool can get the values for the `org.opencontainers.image.source` and `org.opencontainers.image.revision` annotations from git. `org.opencontainers.image.source` is derived from the git origin and `org.opencontainers.image.revision` is the git commit ref. +When building the buildpack, the `buildpack` tool can get the values for the `org.opencontainers.image.source` and `org.opencontainers.image.revision` annotations from git. `org.opencontainers.image.source` is derived from the git origin and `org.opencontainers.image.revision` is the git commit hash. The other annotation values come from `buildpack.toml` mapped to OCI annotations as follows: From d2b2b1a9b196a9e5963454f20a3948e6d8ef7207 Mon Sep 17 00:00:00 2001 From: Craig Andrews Date: Thu, 18 Jul 2024 13:21:57 -0400 Subject: [PATCH 186/222] Consistently use the term `pack` tool instead of `buildpack` tool Signed-off-by: Craig Andrews --- text/0000-oci-image-annotations.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-oci-image-annotations.md b/text/0000-oci-image-annotations.md index a77b0a3ea..a711cb45c 100644 --- a/text/0000-oci-image-annotations.md +++ b/text/0000-oci-image-annotations.md @@ -12,7 +12,7 @@ # Summary [summary]: #summary -The `buildpack` tool should set OCI annotations on the OCI artifacts it produces providing users of these buildpacks with a consistent, standard mechanism for gathering information about the buildpack, including how to find its documentation and what version it is. The annotation values can be gathered from existing data sources (such as `buildpack.toml`) therefore not requiring any additional effort on the part of users of the `buildpack` tool. +The `pack` tool should set OCI annotations on the OCI artifacts it produces providing users of these buildpacks with a consistent, standard mechanism for gathering information about the buildpack, including how to find its documentation and what version it is. The annotation values can be gathered from existing data sources (such as `buildpack.toml`) therefore not requiring any additional effort on the part of users of the `pack` tool. # Definitions [definitions]: #definitions @@ -45,7 +45,7 @@ The target personas as buildpack users, platform operators, and platform impleme # How it Works [how-it-works]: #how-it-works -When building the buildpack, the `buildpack` tool can get the values for the `org.opencontainers.image.source` and `org.opencontainers.image.revision` annotations from git. `org.opencontainers.image.source` is derived from the git origin and `org.opencontainers.image.revision` is the git commit hash. +When building the buildpack, the `pack` tool can get the values for the `org.opencontainers.image.source` and `org.opencontainers.image.revision` annotations from git. `org.opencontainers.image.source` is derived from the git origin and `org.opencontainers.image.revision` is the git commit hash. The other annotation values come from `buildpack.toml` mapped to OCI annotations as follows: @@ -66,7 +66,7 @@ The following example values are from [Paketo Buildpack for Java 13.0.1](https:/ # Migration [migration]: #migration -The `buildpack` tool would be modified to set the annotations. Because the tool _should_ set these annotations (not _must_ set), buildpacks created with earlier versions of the tool are still consider to be valid in accordance with the distribution specification. +The `pack` tool would be modified to set the annotations. Because the tool _should_ set these annotations (not _must_ set), buildpacks created with earlier versions of the tool are still consider to be valid in accordance with the distribution specification. # Drawbacks [drawbacks]: #drawbacks From 438fddec807e2dac65b1f910852eac9a5886f8fe Mon Sep 17 00:00:00 2001 From: Craig Andrews Date: Thu, 18 Jul 2024 13:24:45 -0400 Subject: [PATCH 187/222] Fix incorrect tense of "consider" Co-authored-by: Josh W Lewis Signed-off-by: Craig Andrews --- text/0000-oci-image-annotations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-oci-image-annotations.md b/text/0000-oci-image-annotations.md index a711cb45c..4e1d0642f 100644 --- a/text/0000-oci-image-annotations.md +++ b/text/0000-oci-image-annotations.md @@ -66,7 +66,7 @@ The following example values are from [Paketo Buildpack for Java 13.0.1](https:/ # Migration [migration]: #migration -The `pack` tool would be modified to set the annotations. Because the tool _should_ set these annotations (not _must_ set), buildpacks created with earlier versions of the tool are still consider to be valid in accordance with the distribution specification. +The `pack` tool would be modified to set the annotations. Because the tool _should_ set these annotations (not _must_ set), buildpacks created with earlier versions of the tool are still considered to be valid in accordance with the distribution specification. # Drawbacks [drawbacks]: #drawbacks From cc8e2f4532999664f9f5b2404371c2e4e268fbee Mon Sep 17 00:00:00 2001 From: Craig Andrews Date: Thu, 18 Jul 2024 13:25:21 -0400 Subject: [PATCH 188/222] Use the term "packaging" as that's the term used by the tool Co-authored-by: Josh W Lewis Signed-off-by: Craig Andrews --- text/0000-oci-image-annotations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-oci-image-annotations.md b/text/0000-oci-image-annotations.md index 4e1d0642f..678d5d30e 100644 --- a/text/0000-oci-image-annotations.md +++ b/text/0000-oci-image-annotations.md @@ -45,7 +45,7 @@ The target personas as buildpack users, platform operators, and platform impleme # How it Works [how-it-works]: #how-it-works -When building the buildpack, the `pack` tool can get the values for the `org.opencontainers.image.source` and `org.opencontainers.image.revision` annotations from git. `org.opencontainers.image.source` is derived from the git origin and `org.opencontainers.image.revision` is the git commit hash. +When packaging the buildpack, the `pack` tool can get the values for the `org.opencontainers.image.source` and `org.opencontainers.image.revision` annotations from git. `org.opencontainers.image.source` is derived from the git origin and `org.opencontainers.image.revision` is the git commit hash. The other annotation values come from `buildpack.toml` mapped to OCI annotations as follows: From 9b15d01a7d1ab49eba7bc74a953d2ea469e22f93 Mon Sep 17 00:00:00 2001 From: Enrico Candino Date: Thu, 8 Aug 2024 17:12:54 +0200 Subject: [PATCH 189/222] fixed lifecycle descriptor link Signed-off-by: Enrico Candino --- text/0041-api-version-compat.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0041-api-version-compat.md b/text/0041-api-version-compat.md index ae814167f..252935829 100644 --- a/text/0041-api-version-compat.md +++ b/text/0041-api-version-compat.md @@ -43,7 +43,7 @@ The spec will use future `0.x` API releases to include any desired changes with This RFC does not make changes to the API compatibility rules post 1.0. 1.x APIs version are stil assumed to be purely additive. ## Lifecycle -The lifecycle will use the [lifecycle descriptor](https://github.com/buildpacks/rfcs/blob/pack-publish-buildpack/text/0011-lifecycle-descriptor.md) +The lifecycle will use the [lifecycle descriptor](https://github.com/buildpacks/rfcs/blob/main/text/0011-lifecycle-descriptor.md) to indicate the implemented API versions as before. However, the compatibility assumptions will change. It will be assumed that a lifecycle declaring support for the `0.x` version of the API, supports all 0.2-0.x API versions. From 4c05331f64d4bbc77d50371bc491be5459aa3677 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 8 Aug 2024 10:16:44 -0500 Subject: [PATCH 190/222] RFC 0130 [#314] Signed-off-by: Joe Kutner --- ...i-image-annotations.md => 0130-oci-image-annotations.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-oci-image-annotations.md => 0130-oci-image-annotations.md} (97%) diff --git a/text/0000-oci-image-annotations.md b/text/0130-oci-image-annotations.md similarity index 97% rename from text/0000-oci-image-annotations.md rename to text/0130-oci-image-annotations.md index 678d5d30e..0e5e8f3c1 100644 --- a/text/0000-oci-image-annotations.md +++ b/text/0130-oci-image-annotations.md @@ -3,10 +3,10 @@ - Name: OCI Image Annotations on Buildpacks - Start Date: 2024-06-26 - Author(s): @candrews -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#314](https://github.com/buildpacks/rfcs/pull/314) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: https://github.com/buildpacks/rfcs/issues/318 - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Summary From cb85e957a36d823cc8ce4a940616f2613c2bb512 Mon Sep 17 00:00:00 2001 From: Joe Kutner Date: Thu, 8 Aug 2024 10:24:57 -0500 Subject: [PATCH 191/222] Update merge-rfc.sh script. remove auto issue creation --- merge-rfc.sh | 31 ++++++------------------------- 1 file changed, 6 insertions(+), 25 deletions(-) diff --git a/merge-rfc.sh b/merge-rfc.sh index 8f90848f5..f2c19b34c 100755 --- a/merge-rfc.sh +++ b/merge-rfc.sh @@ -50,13 +50,6 @@ require_command() { require_command git require_command jq -require_command issues-generation - -if [[ -z "${GITHUB_TOKEN:-}" ]]; then - require_command op - echo "> Pulling GitHub token from vault..." - GITHUB_TOKEN=$(op read op://Shared/7xorpxvz3je3vozqg3fy3wrcg4/credential --account buildpacks) -fi #### # INPUTS / VALIDATION @@ -107,25 +100,13 @@ fi RFC_ID=$(generate_id) echo "> Generated RFC number: ${RFC_ID}" -echo "> Creating issues for PR#${PR_NUMBER}" -export GITHUB_TOKEN - -issues-generation create --pr "${OWNER}/${REPO}#${PR_NUMBER}" --bot $BOT_USERNAME --prepend "[RFC #${RFC_ID}] " -ISSUES_TO_LINK=$(issues-generation list --pr "${OWNER}/${REPO}#${PR_NUMBER}" --bot $BOT_USERNAME --json | jq -r '[.[] | select(.num) | .repo + "#" + (.num|tostring) ] | join(" ")') - -for ISSUE in ${ISSUES_TO_LINK}; do - if [[ $ISSUES_TEXT == "N/A" ]]; then - ISSUES_TEXT=$(link_issue "$ISSUE") - else - ISSUES_TEXT+=", $(link_issue "$ISSUE")" - fi -done - - if [[ $NO_ISSUES = false && $ISSUES_TEXT == "N/A" ]]; then - echo -e "ERROR! No issues were provided. Are you sure there are no issues that should be linked?" - echo -e "ERROR! Either -i or -n is required\n" - usage + echo "> Please create an issue by following the link below:" + echo "https://github.com/buildpacks/rfcs/issues/new?assignees=&labels=type%2Ftracking&projects=&template=tracking.md&title=%5BRFC+%23${RFC_ID}%5D+%3C+-+INSERT+RFC+TITLE%3E" + echo "" + read -p "Press Enter to continue" Please enter the issue link: " ISSUES_TEXT fi echo "> Pulling latest changes...." From 1d98e4f55a53b942bf3d3acd4870d2a625667848 Mon Sep 17 00:00:00 2001 From: Rashad Sirajudeen Date: Thu, 15 Feb 2024 11:05:02 +0530 Subject: [PATCH 192/222] Initial draft, up for feedback --- text/0000-my-feature.md | 113 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 text/0000-my-feature.md diff --git a/text/0000-my-feature.md b/text/0000-my-feature.md new file mode 100644 index 000000000..2fdca1454 --- /dev/null +++ b/text/0000-my-feature.md @@ -0,0 +1,113 @@ +# Meta +[meta]: #meta +- Name: Implementing pack detect command +- Start Date: 2024-02-15 +- Author(s): @rashadism +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: N/A + +# Summary +[summary]: #summary + +The `pack detect` command is introduced to the Cloud Native Buildpacks ecosystem, providing a way to run only the detect phase of the buildpack lifecycle. This feature enhances the developer experience by allowing them to quickly determine which buildpacks are relevant for their application without progressing through the entire build process. This was partially discussed in [issue #681](https://github.com/buildpacks/pack/issues/681) but the issue was about implementing a `dry-run` flag. With further discussion with @jjbustamante, decided to go forward with this as a new pack pack command rather than a flag. + +# Definitions +[definitions]: #definitions + +Make a list of the definitions that may be useful for those reviewing. Include phrases and words that buildpack authors or other interested parties may not be familiar with. + +# Motivation +[motivation]: #motivation + +- Simplify and streamline the build process by providing a targeted command for buildpack detection. +- Reduce build times by skipping unnecessary phases of the buildpack lifecycle. +- Enable developers to quickly identify which buildpacks are applicable to their application without waiting for the entire build process to complete, or having to `Ctrl+C` midway through. + +# What it is +[what-it-is]: #what-it-is + +This provides a high level overview of the feature. + +- Define any new terminology. +- Define the target persona: buildpack author, buildpack user, platform operator, platform implementor, and/or project contributor. +- Explaining the feature largely in terms of examples. +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. +- If applicable, describe the differences between teaching this to existing users and new users. + +# How it Works +[how-it-works]: #how-it-works + +Ideally, the user should run something like `pack detect --path ./path/to/project --builder builder:name` and it should run only the detect binary in the lifecycle and output the logs / output of it. Preferrably, can also output the `group.toml` to a directory specified with `--detect-output-dir`. + +The following flags should be supported and they will work more or less like `pack build`. + +|Short|Long|type| +|----|-------------|----| +|-B |--builder|string| +|-b |--buildpack|strings| +|-r |--buildpack-registry|string| +|-e|--env|stringArray| +||--env-file|stringArray| +||--extension|strings| +|-h|--help| +|-p|--path|string| +||--post-buildpack|stringArray| +||--pre-buildpack|stringArray| +||--detect-output-dir|string| + +# Migration +[migration]: #migration + +This feature does not introduce any breaks to public APIs or compatibility. It provides additional functionality within the existing Cloud Native Buildpacks CLI tooling, enhancing the developer experience without requiring changes to existing workflows or configurations. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Alternatives +[alternatives]: #alternatives + +Initially thought of implementing this through something like `pack build --detect`. But after further discussion with @jjbustamante and for the following reasons, decided to do this functionality to a new command. +- The main use case of `pack build` is to create OCI images, and detect is just a binary in the lifecycle, so it doesn't make much sense to include it in there. +- To avoid making the mostly used `pack build` command overly complicated. + +Also, instead of `pack detect`, something like `pack execute --phase detect` can also be done, where the user can select exclusively what phase they need to run. Can start by implementing just the `detect` phase. + +# Prior Art +[prior-art]: #prior-art + +This has been discussed in Issue #681 before, and looked like it was a long awaited feature and currently a few workarounds are being used to get this functionality. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +Fill after initial discussion + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes +Since this is a new command, the functionality of this command will have to be amended to the spec / docs. + +# History +[history]: #history + + \ No newline at end of file From a7c216338203eb51abce7fd18bac7ec86fb1a22d Mon Sep 17 00:00:00 2001 From: Rashad Sirajudeen Date: Fri, 15 Mar 2024 11:35:58 +0530 Subject: [PATCH 193/222] Added reviewed changes Signed-off-by: Rashad Sirajudeen --- text/0000-my-feature.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/text/0000-my-feature.md b/text/0000-my-feature.md index 2fdca1454..a9435ffc8 100644 --- a/text/0000-my-feature.md +++ b/text/0000-my-feature.md @@ -40,7 +40,7 @@ This provides a high level overview of the feature. # How it Works [how-it-works]: #how-it-works -Ideally, the user should run something like `pack detect --path ./path/to/project --builder builder:name` and it should run only the detect binary in the lifecycle and output the logs / output of it. Preferrably, can also output the `group.toml` to a directory specified with `--detect-output-dir`. +Ideally, the user should run something like `pack detect --path ./path/to/project --builder builder:name` and it should run the analyze binary, followed by the detect binary in the lifecycle and output the logs / output of it. Preferrably, can also output the `group.toml` to a directory specified with `--detect-output-dir`. The reason to run the analyze binary is to get information about the run image that may impact the outcome of detect via CNB_TARGET_* environment variables. The following flags should be supported and they will work more or less like `pack build`. @@ -49,14 +49,24 @@ The following flags should be supported and they will work more or less like `pa |-B |--builder|string| |-b |--buildpack|strings| |-r |--buildpack-registry|string| +||--detect-output-dir|string| +|-d|--descriptor|string| +||--docker-host|string| |-e|--env|stringArray| ||--env-file|stringArray| ||--extension|strings| +||--gid|int| |-h|--help| +||--lifecycle-image|string| +||--network|string| |-p|--path|string| ||--post-buildpack|stringArray| ||--pre-buildpack|stringArray| -||--detect-output-dir|string| +||--pull-policy|string| +||--run-image|string| +||--uid|int| +||--workspace|string| + # Migration [migration]: #migration From 3ed07bdf3ec1934f0e468123f3f733f3becef07f Mon Sep 17 00:00:00 2001 From: Rashad Sirajudeen Date: Tue, 14 May 2024 10:03:00 +0530 Subject: [PATCH 194/222] Update text/0000-my-feature.md Co-authored-by: Daniel Mikusa Signed-off-by: Rashad Sirajudeen --- text/0000-my-feature.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-my-feature.md b/text/0000-my-feature.md index a9435ffc8..6a9be7107 100644 --- a/text/0000-my-feature.md +++ b/text/0000-my-feature.md @@ -25,6 +25,7 @@ Make a list of the definitions that may be useful for those reviewing. Include p - Simplify and streamline the build process by providing a targeted command for buildpack detection. - Reduce build times by skipping unnecessary phases of the buildpack lifecycle. - Enable developers to quickly identify which buildpacks are applicable to their application without waiting for the entire build process to complete, or having to `Ctrl+C` midway through. +- Lighter-weight integration testing of the build plan. # What it is [what-it-is]: #what-it-is From 3e8c78ffc63b807737c3dafb2c1d1e4733194f55 Mon Sep 17 00:00:00 2001 From: Rashad Sirajudeen Date: Thu, 13 Jun 2024 19:51:12 +0530 Subject: [PATCH 195/222] fixed typo Signed-off-by: Rashad Sirajudeen --- text/0000-my-feature.md | 69 +++++++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 27 deletions(-) diff --git a/text/0000-my-feature.md b/text/0000-my-feature.md index 6a9be7107..fe465170e 100644 --- a/text/0000-my-feature.md +++ b/text/0000-my-feature.md @@ -1,5 +1,7 @@ # Meta + [meta]: #meta + - Name: Implementing pack detect command - Start Date: 2024-02-15 - Author(s): @rashadism @@ -10,16 +12,19 @@ - Supersedes: N/A # Summary + [summary]: #summary -The `pack detect` command is introduced to the Cloud Native Buildpacks ecosystem, providing a way to run only the detect phase of the buildpack lifecycle. This feature enhances the developer experience by allowing them to quickly determine which buildpacks are relevant for their application without progressing through the entire build process. This was partially discussed in [issue #681](https://github.com/buildpacks/pack/issues/681) but the issue was about implementing a `dry-run` flag. With further discussion with @jjbustamante, decided to go forward with this as a new pack pack command rather than a flag. +The `pack detect` command is introduced to the Cloud Native Buildpacks ecosystem, providing a way to run only the detect phase of the buildpack lifecycle. This feature enhances the developer experience by allowing them to quickly determine which buildpacks are relevant for their application without progressing through the entire build process. This was partially discussed in [issue #681](https://github.com/buildpacks/pack/issues/681) but the issue was about implementing a `dry-run` flag. With further discussion with @jjbustamante, decided to go forward with this as a new `pack detect` command rather than a flag. # Definitions + [definitions]: #definitions Make a list of the definitions that may be useful for those reviewing. Include phrases and words that buildpack authors or other interested parties may not be familiar with. # Motivation + [motivation]: #motivation - Simplify and streamline the build process by providing a targeted command for buildpack detection. @@ -28,6 +33,7 @@ Make a list of the definitions that may be useful for those reviewing. Include p - Lighter-weight integration testing of the build plan. # What it is + [what-it-is]: #what-it-is This provides a high level overview of the feature. @@ -39,70 +45,79 @@ This provides a high level overview of the feature. - If applicable, describe the differences between teaching this to existing users and new users. # How it Works + [how-it-works]: #how-it-works -Ideally, the user should run something like `pack detect --path ./path/to/project --builder builder:name` and it should run the analyze binary, followed by the detect binary in the lifecycle and output the logs / output of it. Preferrably, can also output the `group.toml` to a directory specified with `--detect-output-dir`. The reason to run the analyze binary is to get information about the run image that may impact the outcome of detect via CNB_TARGET_* environment variables. +Ideally, the user should run something like `pack detect --path ./path/to/project --builder builder:name` and it should run the analyze binary, followed by the detect binary in the lifecycle and output the logs / output of it. Preferrably, can also output the `group.toml` to a directory specified with `--detect-output-dir`. The reason to run the analyze binary is to get information about the run image that may impact the outcome of detect via CNB*TARGET*\* environment variables. The following flags should be supported and they will work more or less like `pack build`. -|Short|Long|type| -|----|-------------|----| -|-B |--builder|string| -|-b |--buildpack|strings| -|-r |--buildpack-registry|string| -||--detect-output-dir|string| -|-d|--descriptor|string| -||--docker-host|string| -|-e|--env|stringArray| -||--env-file|stringArray| -||--extension|strings| -||--gid|int| -|-h|--help| -||--lifecycle-image|string| -||--network|string| -|-p|--path|string| -||--post-buildpack|stringArray| -||--pre-buildpack|stringArray| -||--pull-policy|string| -||--run-image|string| -||--uid|int| -||--workspace|string| - +| Short | Long | type | +| ----- | -------------------- | ----------- | +| -B | --builder | string | +| -b | --buildpack | strings | +| -r | --buildpack-registry | string | +| | --detect-output-dir | string | +| -d | --descriptor | string | +| | --docker-host | string | +| -e | --env | stringArray | +| | --env-file | stringArray | +| | --extension | strings | +| | --gid | int | +| -h | --help | +| | --lifecycle-image | string | +| | --network | string | +| -p | --path | string | +| | --post-buildpack | stringArray | +| | --pre-buildpack | stringArray | +| | --pull-policy | string | +| | --run-image | string | +| | --uid | int | +| | --workspace | string | # Migration + [migration]: #migration This feature does not introduce any breaks to public APIs or compatibility. It provides additional functionality within the existing Cloud Native Buildpacks CLI tooling, enhancing the developer experience without requiring changes to existing workflows or configurations. # Drawbacks + [drawbacks]: #drawbacks -Why should we *not* do this? +Why should we _not_ do this? # Alternatives + [alternatives]: #alternatives Initially thought of implementing this through something like `pack build --detect`. But after further discussion with @jjbustamante and for the following reasons, decided to do this functionality to a new command. + - The main use case of `pack build` is to create OCI images, and detect is just a binary in the lifecycle, so it doesn't make much sense to include it in there. - To avoid making the mostly used `pack build` command overly complicated. Also, instead of `pack detect`, something like `pack execute --phase detect` can also be done, where the user can select exclusively what phase they need to run. Can start by implementing just the `detect` phase. # Prior Art + [prior-art]: #prior-art This has been discussed in Issue #681 before, and looked like it was a long awaited feature and currently a few workarounds are being used to get this functionality. # Unresolved Questions + [unresolved-questions]: #unresolved-questions Fill after initial discussion # Spec. Changes (OPTIONAL) + [spec-changes]: #spec-changes + Since this is a new command, the functionality of this command will have to be amended to the spec / docs. # History + [history]: #history \ No newline at end of file +---> From 572bbcd8d4abd68719b155d928db83f01121b3e5 Mon Sep 17 00:00:00 2001 From: Rashad Sirajudeen Date: Fri, 14 Jun 2024 10:56:13 +0530 Subject: [PATCH 196/222] Updated with changes from feedback Signed-off-by: Rashad Sirajudeen --- text/0000-my-feature.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/text/0000-my-feature.md b/text/0000-my-feature.md index fe465170e..085539aa1 100644 --- a/text/0000-my-feature.md +++ b/text/0000-my-feature.md @@ -15,7 +15,7 @@ [summary]: #summary -The `pack detect` command is introduced to the Cloud Native Buildpacks ecosystem, providing a way to run only the detect phase of the buildpack lifecycle. This feature enhances the developer experience by allowing them to quickly determine which buildpacks are relevant for their application without progressing through the entire build process. This was partially discussed in [issue #681](https://github.com/buildpacks/pack/issues/681) but the issue was about implementing a `dry-run` flag. With further discussion with @jjbustamante, decided to go forward with this as a new `pack detect` command rather than a flag. +The `pack execute` command is introduced to the Cloud Native Buildpacks ecosystem, providing a way to run only selected phases of the buildpack lifecycle. For the moment, only `detect` will be implemented, which will be invoked through `pack execute detect`.This feature enhances the developer experience by allowing them to quickly determine which buildpacks are relevant for their application without progressing through the entire build process. This was partially discussed in [issue #681](https://github.com/buildpacks/pack/issues/681) but the issue was about implementing a `dry-run` flag. With further discussion with @jjbustamante, decided to go forward with this as a new `pack detect` command rather than a flag, and after further review it will be implemented as `pack execute detect`. # Definitions @@ -27,6 +27,7 @@ Make a list of the definitions that may be useful for those reviewing. Include p [motivation]: #motivation +- Enable the running of selected phases of buildpacks upon need. - Simplify and streamline the build process by providing a targeted command for buildpack detection. - Reduce build times by skipping unnecessary phases of the buildpack lifecycle. - Enable developers to quickly identify which buildpacks are applicable to their application without waiting for the entire build process to complete, or having to `Ctrl+C` midway through. @@ -48,7 +49,7 @@ This provides a high level overview of the feature. [how-it-works]: #how-it-works -Ideally, the user should run something like `pack detect --path ./path/to/project --builder builder:name` and it should run the analyze binary, followed by the detect binary in the lifecycle and output the logs / output of it. Preferrably, can also output the `group.toml` to a directory specified with `--detect-output-dir`. The reason to run the analyze binary is to get information about the run image that may impact the outcome of detect via CNB*TARGET*\* environment variables. +Ideally, the user should run something like `pack execute detect --path ./path/to/project --builder builder:name` and it should run the analyze binary, followed by the detect binary in the lifecycle and output the logs / output of it. This also copies `group.toml` to a directory specified with `--detect-output-dir`, if the flag was enabled. The reason to run the analyze binary is to get information about the run image that may impact the outcome of detect via CNB*TARGET*\* environment variables. The following flags should be supported and they will work more or less like `pack build`. @@ -91,13 +92,11 @@ Why should we _not_ do this? [alternatives]: #alternatives -Initially thought of implementing this through something like `pack build --detect`. But after further discussion with @jjbustamante and for the following reasons, decided to do this functionality to a new command. +Initially thought of implementing this through something like `pack build --detect`. But after further discussion with @jjbustamante and for the following reasons, decided to do this functionality to a new command. Upon further review, this will be implemented as `pack execute detect ..` - The main use case of `pack build` is to create OCI images, and detect is just a binary in the lifecycle, so it doesn't make much sense to include it in there. - To avoid making the mostly used `pack build` command overly complicated. -Also, instead of `pack detect`, something like `pack execute --phase detect` can also be done, where the user can select exclusively what phase they need to run. Can start by implementing just the `detect` phase. - # Prior Art [prior-art]: #prior-art From 4f5e23ba18dabe17e51ce40bab3bf6ef5140ecd1 Mon Sep 17 00:00:00 2001 From: Josh W Lewis Date: Fri, 6 Oct 2023 08:58:28 -0500 Subject: [PATCH 197/222] RFC: Build observability using OpenTelemetry tracing Signed-off-by: Josh W Lewis Update telemetry file location Signed-off-by: Josh W Lewis Ad drawback about /layers/tracing Signed-off-by: Josh W Lewis --- text/0000-build-observability.md | 411 +++++++++++++++++++++++++++++++ 1 file changed, 411 insertions(+) create mode 100644 text/0000-build-observability.md diff --git a/text/0000-build-observability.md b/text/0000-build-observability.md new file mode 100644 index 000000000..09cf2f940 --- /dev/null +++ b/text/0000-build-observability.md @@ -0,0 +1,411 @@ +# Meta +[meta]: #meta +- Name: Buildpack Observability +- Start Date: 2022-10-05 +- Author(s): @joshwlewis +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +This RFC proposes leveraging [OpenTelemetry](https://opentelemetry.io/) to +grant platform operators and buildpack operators more insight into buildpack +performance and behavior. This RFC describes new opt-in functionality +for pack, the lifecycle, and the buildpack spec such that OpenTelemetry data may be +exported to the build file system. + +# Definitions +[definitions]: #definitions + +- [OpenTelemetry](https://opentelemetry.io/): A collection of APIs, SDKs, and tools that can be used it to instrument, generate, collect, and export telemetry data. +- [Traces](https://opentelemetry.io/docs/concepts/signals/traces/): Telemetry + category that describes the path of software execution. + + +# Motivation +[motivation]: #motivation + +Buildpack authors and platform operators desire insight into usage, error +scenarios, and performance of builds and buildpacks on their platform. The +following questions are all important for these folks, but difficult to answer: + +- "Which buildpacks commonly fail to compile?" +- "How often does a particular error scenario occur?" +- "How long does each buildpack compile phase take?" +- "How often is a certain buildpack used?" +- "Which versions of Go are being installed?" +- "How long does it take to download node_modules?" + +Instrumenting lifecycle and buildpacks with opt-in OpenTelemetry tracing will +allow platform operators to better understand performance and behavior of their +builds and buildpacks and as a result, provide better service and build +experiences. + +To protect privacy and prevent unnecessary collection of data, this +functionality shall be optional and anonymous. + +# What it is +[what-it-is]: #what-it-is + +This RFC aims to provide a solution for two types of OpenTelemetry traces: + +1) Lifecycle tracing: Buildpack-agnostic trace data like which buildpacks were +available, which buildpacks were detected, how long the detect, build, or +export phase took, and so on. This telemetry data may be exported by lifecycle. +2) Buildpack tracing: Telemetry data specific to a buildpack like how long it +took to download a language binary, which language version was selected, and so +on. This telemetry data may be exported by buildpacks. + +Though the sources and contents of the telemetry data differ, both types may +be emitted to the build file system in OpenTelemetry's [File Exporter +Format](https://opentelemetry.io/docs/specs/otel/protocol/file-exporter/). + +In this solution, each lifecycle phase would write a `.jsonl` file with +tracing data for that phase. For example, `lifecycle detector --telemetry` +would write to `/layers/tracing/lifecycle/detect.jsonl`. Additionally each +buildpack may also write tracing data to it's own `.jsonl` files (at +`/layers/tracing/buildpacks/#{id}@#{version}-#{phase}.jsonl`). + +These `.jsonl` files may be read by platform operators for consumption, +transformation, enrichment, and/or export to an OpenTelemetry backend. Given +that builds may crash or fail at any point, these files must be written to +often and regularly to prevent data loss. + +Platform operators will likely want to view or analyze this data. These +telemetry files are in OTLP compatible format, so may be exported to one or +more OpenTelemetry backends like Honeycomb, Prometheus, and [many +others](https://opentelemetry.io/ecosystem/vendors/). + +Additionally, these traces may be correlated with traces in a platform +operator's system via context propagation. The `CNB_OTEL_TRACEPARENT` may be +provided by a platform to the build environment, such that generated traces +inherit `trace-id` and `parent-id` from platform systems. + +# How it Works +[how-it-works]: #how-it-works + +### Lifecycle telemetry files + +If `lifecycle` is provided the telemetry opt-in flag (such as `--telemetry`), +`lifecycle` phases (such as `detect`, `build`, `export`) may emit an +OpenTelemetry File Export with tracing data to a known location, such as +`/layers/tracing/lifecycle/detect.jsonl` with contents like this: + +```json +{ + "resourceSpans": [ + { + "resource": { + "attributes": [ + { + "key": "lifecycle.version", + "value": { + "stringValue": "0.17.1" + } + } + ] + }, + "scopeSpans": [ + { + "scope": {}, + "spans": [ + { + "traceId": "", + "spanId": "", + "parentSpanId": "", + "name": "buildpack-detect", + "startTimeUnixNano": "1581452772000000321", + "endTimeUnixNano": "1581452773000000789", + "droppedAttributesCount": 2, + "events": [ + { + "timeUnixNano": "1581452773000000123", + "name": "detect-pass" + } + ], + "attributes": [ + { + "key": "buildpack-id", + "value": { + "stringValue": "heroku/nodejs-engine" + } + } + ], + "droppedEventsCount": 1 + } + ] + } + ] + } + ] +} +``` + + +### Buildpack telemetry files + +During a buildpack's `detect` and/or `build` execution, a buildpack may emit +an OpenTelemetry File Export with tracing data to `/layers/tracing/buildpacks/#{id}@#{version}-#{phase}.jsonl` +with contents like this: + +```json +{ + "resourceSpans": [ + { + "resource": { + "attributes": [ + { + "key": "lifecycle.version", + "value": { + "stringValue": "0.17.1" + } + } + ] + }, + "scopeSpans": [ + { + "scope": {}, + "spans": [ + { + "traceId": "", + "spanId": "", + "parentSpanId": "", + "name": "buildpack-detect", + "startTimeUnixNano": "1581452772000000321", + "endTimeUnixNano": "1581452773000000789", + "droppedAttributesCount": 2, + "events": [ + { + "timeUnixNano": "1581452773000000123", + "name": "detect-pass" + } + ], + "attributes": [ + { + "key": "buildpack-id", + "value": { + "stringValue": "heroku/nodejs-engine" + } + } + ], + "droppedEventsCount": 1 + } + ] + } + ] + } + ] +} +``` + +### Location + +All tracing files should be written to `/layers/tracing/`. Lifecycle execution +traces should be written to `/layers/tracing/lifecycle/{phase}.jsonl`. +Buildpack traces may be written to +`/layers/tracing/buildpacks/{id}@{version}-{phase}.jsonl`. +Extension traces may be written to +`/layers/tracing/extensions/{id}@{version}-{phase}.jsonl`. + +A completed build with tracing might have a tracing file hierarchy like this: + +``` + +└── tracing + ├── buildpacks + │ ├── other-id@other-version-detect.jsonl + │ ├── some-id@some-version-build.jsonl + │ └── some-id@some-version-detect.jsonl + ├── extensions + │ ├── some-id@some-version-detect.jsonl + │ └── some-id@some-version-generate.jsonl + └── lifecycle + ├── analyze.jsonl + ├── build.jsonl + ├── detect.jsonl + ├── export.jsonl + ├── extend.jsonl + └── restore.jsonl +``` + +### Lifetime + +Telemetry files may be written at any point during the build, so that they +are persisted in cases of failures to detect, failures to build, process +terminations, or crashes. The `jsonl` format allows telemetry libraries to +safely append additional json objects to the end of a telemetry file, so +telemetry data can be flushed to the file frequently. Telemetry files should +not be truncated or deleted so that telemetry processing by a platform can +happen during or after a build. Telemetry files should not be included in the +build result, as they are not relevant, and would likely negatively impact +image size and reproduceability. + +### Access + +The telemetry files should be group readable so that they may be analyzed by +the user and/or platform during and/or after the build. The telemetry files +must also be group readable so that buildpacks and lifecycle can write to them, +but buildpacks and the lifecycle shall read and write only their own files. + +### Context Propagation + +To allow correlation of lifecycle and buildpack traces to traces in platform +operator's systems, `CNB_OTEL_TRACEPARENT` may be provided for `lifecycle` and +buildpacks. The value of this env var should follow +[W3C Trace Context specification for traceparent field values](https://www.w3.org/TR/trace-context/#traceparent-header-field-values). +If provided, generated traces by lifecycle and buildpacks shall inherit the +`trace-id` and `parent-id` provided therein. + +### Consumption + +This RFC leaves the consumption of telemetry files to the platform operator. +Platform operators choosing to use these metrics may read them either during +or after the build. This can be done using existing OpenTelemetry libraries. +Platform operators may choose to optionally enrich or modify the tracing data +as they see fit (with data like `instance_id` or `build_id`). Platform +operators will likely want to export this data to an OpenTelemetry backend for +persistence and analysis, and again, this may be done with existing +OpenTelemetry libraries. + +### Viewing and Analyzing + +Once the lifecycle and buildpack traces are exported to an OpenTelemetry +backend, platform operators should be able to (depending on the features of the +backend): + +- View the complete trace for a build +- View or query attributes attached to spans (e.g. `buildpack_id`, + `nodejs_version`) +- View or query span durations +- View or query error types and/or messages +- and more + +### Intent + +The purpose and intent of these files is to provide anonymous build +observability data for users and platform operators. These files shall not +be used for other intents. For example: + +- These files shall not be used as an API, contract, or communication mechanism + between buildpacks. +- These files shall not record any personally identifiable information (such + as usernames, email addresses, IP Addresses, etc.). +- These files shall not record any potentially business sensitive information + (such as passwords, access keys, resulting image name and/or urls, source + code repository name, etc.). + +# Migration +[migration]: #migration + +No migration neccessary, this is net-new functionality with no backwards +compatibilty concerns. + +# Drawbacks +[drawbacks]: #drawbacks + +### Privacy Concerns + +This RFC outlines functionality that could be percieved as user tracking. To +help remediate those concerns, these are some factors to remember about this +design: + +1) This functionality is opt-in. `lifecycle` and `pack` will not emit telemetry + data unless the `--telemetry` flag is used. +2) This functionality emits telemetry data only to the build file system. For + `pack` users, the telemetry files are stored in docker volumes on the local + machine. Neither `pack` nor `lifecycle` will "phone home" with telemety data. +3) Neither `pack` nor `lifecycle` collect personally identifiable information + (no emails, usernames, passwords, IP addresses, etc.). + +### File Export Format Status + +While the [File Exporter +Format](https://opentelemetry.io/docs/specs/otel/protocol/file-exporter/) is +an official format, and matches the OTLP format nearly exactly (and thus seems +unlikely to change), it is listed as experimental status. + + +### Additional restricted layer name + +This RFC introduces `/layers/tracing/`. This means that buildpack authors will +be unable to use this directory as a , and lifecycle will need to +prevent usage of this directory. + +# Alternatives +[alternatives]: #alternatives + +### OpenTelemetry Metrics + +[Metrics](https://opentelemetry.io/docs/concepts/signals/metrics/) are another +category of telemetry data that could be used to answer questions about +build and buildpack behavior and performance. However, metrics are intended to +provide statistical information in aggregate. Since `lifecycle` and `pack` +only run one build at a time, there is no way to aggregate information about +multiple builds in `pack` or `lifecycle`. + +### OTLP + +The [OpenTelemetryProtocol](https://opentelemetry.io/docs/specs/otlp/) is a +network delivery protocol for OpenTelemetry data. Instead of emitting files as +this RFC describes, lifecycle and buildpacks could instead connect to an +OpenTelemetry collector provided by the platform operator. This pattern is +well supported and well known. + +However, there are drawbacks: + +- In local `pack build` scenarios, it's unlikely that users would have an + OpenTelemetry collector running. This RFC solution does not require a + collector. +- lifecycle and buildpacks would need to know where the OpenTelemetry collector + is and how to authenticate with it. Lifecycle and buildpacks that wish to + emit telemetry may not want to deal with the mountain of configuration to + support various collectors. +- Platform operators may have complex network topology that may make supporting + this feature challenging (e.g. a firewall between lifecycle and the collector + may still be perceived as a lifecycle malfunction). + +There is an [RFC for this alternative](https://github.com/buildpacks/rfcs/pull/300). + +# Prior Art +[prior-art]: #prior-art + + +- [Feature Request](https://github.com/buildpacks/lifecycle/issues/1208) +- [Slack + Discussion](https://cloud-native.slack.com/archives/C033DV8D9FB/p1695144574408979) +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +Buildpack tracing file locations and format should be added to the [buildpack +specification](https://github.com/buildpacks/spec/blob/main/buildpack.md#build). + +# History +[history]: #history + + From ebfe735b3a853a3fd1b28c451f953f1ec72105b8 Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Thu, 22 Aug 2024 09:11:54 -0500 Subject: [PATCH 198/222] RFC 0131 [#299] Signed-off-by: Jesse Brown --- ...0-build-observability.md => 0131-build-observability.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-build-observability.md => 0131-build-observability.md} (99%) diff --git a/text/0000-build-observability.md b/text/0131-build-observability.md similarity index 99% rename from text/0000-build-observability.md rename to text/0131-build-observability.md index 09cf2f940..235ce5798 100644 --- a/text/0000-build-observability.md +++ b/text/0131-build-observability.md @@ -3,10 +3,10 @@ - Name: Buildpack Observability - Start Date: 2022-10-05 - Author(s): @joshwlewis -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#299](https://github.com/buildpacks/rfcs/pull/299) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: N/A - Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) # Summary From 3fa2a8b9e2f0fc973e4d2cc59a82770c9aabc3bd Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Thu, 22 Aug 2024 09:18:29 -0500 Subject: [PATCH 199/222] Fixing bad merge Signed-off-by: Jesse Brown Signed-off-by: Jesse Brown --- text/{0000-my-feature.md => 0132-implementing-pack-detect.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-my-feature.md => 0132-implementing-pack-detect.md} (100%) diff --git a/text/0000-my-feature.md b/text/0132-implementing-pack-detect.md similarity index 100% rename from text/0000-my-feature.md rename to text/0132-implementing-pack-detect.md From ecb271164b88682e999859346989b449e07a34a0 Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Sat, 5 Oct 2024 10:06:35 +0100 Subject: [PATCH 200/222] Update username (#319) * Update username Signed-off-by: Sambhav Kothari * Update auto-assign-learning.yml Signed-off-by: Sambhav Kothari --------- Signed-off-by: Sambhav Kothari --- .github/auto-assign-bat.yml | 2 +- .github/auto-assign-learning.yml | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/auto-assign-bat.yml b/.github/auto-assign-bat.yml index f61b0b51a..d7a432d1b 100644 --- a/.github/auto-assign-bat.yml +++ b/.github/auto-assign-bat.yml @@ -8,4 +8,4 @@ filterLabels: assignees: - ekcasey - hone -- samj1912 +- sambhav diff --git a/.github/auto-assign-learning.yml b/.github/auto-assign-learning.yml index ca2e089f1..08e981ca5 100644 --- a/.github/auto-assign-learning.yml +++ b/.github/auto-assign-learning.yml @@ -8,4 +8,3 @@ filterLabels: assignees: - jkutner - jromero -- samj1912 From 9efea82cc3023dae597f7d1593f928b6a686931b Mon Sep 17 00:00:00 2001 From: Aidan Delaney Date: Thu, 6 Feb 2025 15:19:18 +0000 Subject: [PATCH 201/222] Provide correct metadata for Windows Containers (#324) Update the RFC number and link to the pull request --- ...s-support.md => 0133-remove-windows-containers-support.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename text/{0000-remove-windows-containers-support.md => 0133-remove-windows-containers-support.md} (95%) diff --git a/text/0000-remove-windows-containers-support.md b/text/0133-remove-windows-containers-support.md similarity index 95% rename from text/0000-remove-windows-containers-support.md rename to text/0133-remove-windows-containers-support.md index eb988ecf4..efeccfd8b 100644 --- a/text/0000-remove-windows-containers-support.md +++ b/text/0133-remove-windows-containers-support.md @@ -3,8 +3,8 @@ - Name: Remove Windows Containers Support - Start Date: 2024-04-11 - Author(s): aidan.delaney -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#311](https://github.com/buildpacks/rfcs/pull/311) - CNB Pull Request: (leave blank) - CNB Issue: (leave blank) - Supersedes: "N/A" From 06afae263877419341eca7046890f26088678cc9 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 1 Feb 2023 02:09:24 -0600 Subject: [PATCH 202/222] RFC: Execution Environments Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 185 ++++++++++++++++++++++++++++ 1 file changed, 185 insertions(+) create mode 100644 text/0000-execution-environments.md diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md new file mode 100644 index 000000000..5b576a874 --- /dev/null +++ b/text/0000-execution-environments.md @@ -0,0 +1,185 @@ +# Meta +[meta]: #meta +- Name: Execution Environments +- Start Date: 2023-01-11 +- Author(s): @hone +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +Add support for different execution environments for buildpacks, with testing as the first use case. + +# Definitions +[definitions]: #definitions + +- Execution Environment - The target environment the OCI image is expected to be run in, i.e. production, test, development. + +# Motivation +[motivation]: #motivation + +The main way Buildpacks are being used is building production images, but this is only one piece of the CI/CD pipeline. Without a solid buildpack test environment story, users will be required alternatives for building their environment. A testing environment, while different, shares many of the same broad strokes ultimately producing an execution environment. This is something Buildpacks are well suited to solve. + +# What it is +[what-it-is]: #what-it-is + +## Test Environments + +One of the goals in this design is to minimize the changes needed. The Buildpack lifecycle is already well suited to produce environments: + +``` +App Source -> OCI Image (Production Environment) -> Execute (Launch Process) +``` + +To create a test environment, it can follow the same flow: + +``` +App Source -> OCI Image (Test Environment) -> Execute (Test Process) +``` + +### Division of Responsibility +With the test OCI Image, a platform can execute the tests in the pipeline as they see fit. This means a bulk of the responsibilities are platform concerns: + +- Set which environment to build for +- Decide which buildpacks to execute +- How to execute the tests +- What is the test result format like [TAP](https://en.wikipedia.org/wiki/Test_Anything_Protocol)? +- How to process the test results +- What to do with the results + +This narrows the scope of the Buildpack responsibilities to what it's already good at: + +- Produce Test Environment +- How to launch the process + - launch.toml with test process marked as default (recommendation) + +## Setting the Execution Enviroment for Build + Buildpack +A platform will set the `CNB_EXEC_ENV` env var to the execution environment desired. Buildpacks can than read this env var to branch or switch on logic needed based on the execution environment. + +In addition, Builder Authors, Buildpack Authors, and App Developers will be able to specify various options to a specific execution enviroment using the `exec-env` key. + +## Development Environments +The specifics of creating development enviroments are out of scope of this RFC, but it's not hard to extrapolate how these kind of changes can assist in creating Buildpacks for development environments. + +# How it Works +[how-it-works]: #how-it-works + +## `exec-env` key in TOML + +In order to support additional execution environments an `exec-env` key will be added to various TOML tables in the project. The value can be any string with `all` having special meaning. `all` will apply to all execution environments and will be the default if not specified. This should make it backwards compatible and optional. When `exec-env` is not set to `all`, the table settings will only be applied to that execution environment. + +### Project Descriptor - `project.toml` (App Developers) + +This file be extended by adding `exec-env` to the following tables: + +`[[io.buildpacks.group]]` +`[[io.buildpacks.pre.group]]` +`[[io.buildpacks.post.group.env]]` +`[[io.buildpacks.build.env]]` + +### `builder.toml` (Builder Authors) + +The only table that `exec-env` will be added to is `[[order.group.env]]`. + +### `buildpack.toml` (Buildpack Authors) + +The only table that `exec-env` will be added to is `[[buildpack.order.group]]`. This only is applicable for composite buildpacks. + +## `CNB_EXEC_ENV` Environment Variable + +This env var will reserve the following values: + +* production +* test +* development + +### Buildpack API + +A buildpack author will be able to determine the execution environment their buildpack is expected to build for by reading the `CNB_EXEC_ENV` environment variable. If this value is not set, a Buildpack Author can assume it's set to `production`. + +### Platform API + +It will be up to the platform to set the environment variable `CNB_EXEC_ENV`. If this value is set, `lifecycle` MUST NOT override this value. If the value is not set, `lifecycle` will set it to `production`. + +# Migration +[migration]: #migration + +In order to allow the Buildpack API and Platform API to be updated independently, both APIs will have a fallback to the `production` value. For `1.0`, it would be nice to drop this behavior and assume it will always be set. + +This also touches Project Descriptor, but since the platform controls the Platform API and the Project Descriptor Extension API it shouldn't cause any issues. + +# Drawbacks +[drawbacks]: #drawbacks + +While the design tries to stay simple, it still adds new additions to the spec. + +* env var to Buildpack API +* env var to Platform API +* field to Project Descriptor, `builder.toml`, and `buildpack.toml`. + +# Alternatives +[alternatives]: #alternatives + +## Separate Execution Environment Project Descriptor files + +When using `Dockerfile`, it's common to create a separate one for other execution environments like `Dockerfile.test`. This makes the file clean and easy to read, but comes at the cost of duplicating setup/code. + +With Project Descriptor, this would remove the need for the `exec-env` key at the cost of replicating buildpack groups. It does bring a big benefit by easily unlocking the `builder` key in Project Descriptor to be specific to an execution environment. This would apply to any fields (not tables) that exist at the `[io.buildpacks]` level. There are some other options if this is desired, where a new table could be created for hoisting fields into that would allow us to add the `exec-env` field. + +# Prior Art +[prior-art]: #prior-art + +## [Heroku Testpack API](https://devcenter.heroku.com/articles/testpack-api) + +As part of the classic buildpack design, Heroku extended the API to include support for tests. It added a `bin/test-compile` phase, which is like the normal `bin/compile` phase but specific for setting things up for test. `bin/test` was called for executing tests. This design let a buildpack have a specific codepath for tests and also do intelligent logic for executing tests based on the source code. For example [in Ruby](https://github.com/heroku/heroku-buildpack-ruby/blob/main/bin/support/ruby_test#L62-L76), the buildpack author can run `bin/rspec` if it detects `rspec`, `bin/rails test` when using rails, or default to `rake test`. + +There were some flaws in this design. Though it's clean to separate production and test code paths, they end up sharing a lot of code. Many of the bash based Heroku buildpacks would just [call `bin/compile`](https://github.com/heroku/heroku-buildpack-nodejs/blob/main/bin/test-compile#L24) with different parameters/env vars. + +## [GOOGLE_DEVMODE](https://cloud.google.com/docs/buildpacks/service-specific-configs#google_devmode) + +These are specific to the Google Cloud Buildpacks for setting a development code path to work with skaffold. + +## [Develop API](https://github.com/buildpacks/spec/pull/71) + +The original Cloud Native Buildpacks spec included a Develop API, but it was never implemented. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- "env" is overloaded as a word since we also use it for environment variables. Is there a better word here? +- Should there be builders that are specific to an execution environment? +- Should the execution environments be an enum or flexible as a string? + - enums will help encourage standardization across buildpacks and platforms. + - strings can help account for use cases we haven't thought of yet. +- Should buildpacks be allowed specify allowlist execution environments? +- What changes are needed in the buildpack registry? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +See ["How it Works"](#how-it-works). + +# History +[history]: #history + + From 8869fb45ab5b7a52f08b29d57207826d4d1bcc32 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 1 Feb 2023 15:04:26 -0600 Subject: [PATCH 203/222] add project descriptor example Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 40 +++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 5b576a874..af11c77da 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -81,6 +81,46 @@ This file be extended by adding `exec-env` to the following tables: `[[io.buildpacks.post.group.env]]` `[[io.buildpacks.build.env]]` +An example would look like this: + +```TOML +[_] +schema-version = "0.3" + +[[io.buildpacks.group]] +id = "buildpacks/ruby" +version = "latest" + +[[io.buildpacks.group]] +id = "buildpacks/metrics-agent" +version = "latest" +exec-env = "production" + +[[io.buildpacks.group]] +id = "buildpacks/headless-chrome" +version = "latest" +exec-env = "test" + +[[io.buildpacks.post.group]] +id = "buildpacks/procfile" +version = "latest" + +[[io.buildpacks.buildenv]] +name = "RAILS_ENV" +value = "production" +exec-env = "production" + +[[io.buildpacks.build.env]] +name = "RAILS_ENV" +value = "test" +exec-env = "test" + +[[io.buildpacks.build.env]] +name = "PARALLEL_WORKERS" +value = "4" +exec-env = "test" +``` + ### `builder.toml` (Builder Authors) The only table that `exec-env` will be added to is `[[order.group.env]]`. From aef70b0cb9ac4a664f4f80857df55b2d4c94bca5 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 1 Feb 2023 15:05:51 -0600 Subject: [PATCH 204/222] highlight include/exclude Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index af11c77da..579b2bec5 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -168,7 +168,7 @@ While the design tries to stay simple, it still adds new additions to the spec. When using `Dockerfile`, it's common to create a separate one for other execution environments like `Dockerfile.test`. This makes the file clean and easy to read, but comes at the cost of duplicating setup/code. -With Project Descriptor, this would remove the need for the `exec-env` key at the cost of replicating buildpack groups. It does bring a big benefit by easily unlocking the `builder` key in Project Descriptor to be specific to an execution environment. This would apply to any fields (not tables) that exist at the `[io.buildpacks]` level. There are some other options if this is desired, where a new table could be created for hoisting fields into that would allow us to add the `exec-env` field. +With Project Descriptor, this would remove the need for the `exec-env` key at the cost of replicating buildpack groups. It does bring a big benefit by easily unlocking the `builder` key in Project Descriptor to be specific to an execution environment. This would apply to any fields (not tables) that exist at the `[io.buildpacks]` level like `include`/`exclude`. There are some other options if this is desired, where a new table could be created for hoisting fields into that would allow us to add the `exec-env` field. # Prior Art [prior-art]: #prior-art @@ -191,7 +191,7 @@ The original Cloud Native Buildpacks spec included a Develop API, but it was nev [unresolved-questions]: #unresolved-questions - "env" is overloaded as a word since we also use it for environment variables. Is there a better word here? -- Should there be builders that are specific to an execution environment? +- Should there be builders that are specific to an execution environment? What about `include` or `exclude`? - Should the execution environments be an enum or flexible as a string? - enums will help encourage standardization across buildpacks and platforms. - strings can help account for use cases we haven't thought of yet. From 4d89ced679f5843ebcb904015653a75f19710f6a Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 12 Apr 2023 13:37:50 -0700 Subject: [PATCH 205/222] Update text/0000-execution-environments.md Co-authored-by: Natalie Arellano Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 579b2bec5..9df603208 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -105,7 +105,7 @@ exec-env = "test" id = "buildpacks/procfile" version = "latest" -[[io.buildpacks.buildenv]] +[[io.buildpacks.build.env]] name = "RAILS_ENV" value = "production" exec-env = "production" From 607afa5b35b8c3357ffe56efdf941b729979d86f Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Wed, 12 Apr 2023 13:37:57 -0700 Subject: [PATCH 206/222] Update text/0000-execution-environments.md Co-authored-by: Josh W Lewis Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 9df603208..e500c4377 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -22,7 +22,7 @@ Add support for different execution environments for buildpacks, with testing as # Motivation [motivation]: #motivation -The main way Buildpacks are being used is building production images, but this is only one piece of the CI/CD pipeline. Without a solid buildpack test environment story, users will be required alternatives for building their environment. A testing environment, while different, shares many of the same broad strokes ultimately producing an execution environment. This is something Buildpacks are well suited to solve. +The main way Buildpacks are being used is building production images, but this is only one piece of the software development process. Without a solid buildpack test environment story, users will be required alternatives for building their environment. A testing environment, while different, shares many of the same broad strokes ultimately producing an execution environment. This is something Buildpacks are well suited to solve. # What it is [what-it-is]: #what-it-is From 9c5a4a2572877e297ac8e456a4ed5543a286b463 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 2 Jan 2025 16:37:50 -0600 Subject: [PATCH 207/222] Reviving this RFC with updates from my discussion with @jabrown85. * changed `exec-env` key to an array * clarified how processes will work * added an open question around reserved string values being namespaced * added some usage clarifications Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 39 +++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index e500c4377..837c0243d 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -22,7 +22,7 @@ Add support for different execution environments for buildpacks, with testing as # Motivation [motivation]: #motivation -The main way Buildpacks are being used is building production images, but this is only one piece of the software development process. Without a solid buildpack test environment story, users will be required alternatives for building their environment. A testing environment, while different, shares many of the same broad strokes ultimately producing an execution environment. This is something Buildpacks are well suited to solve. +The main way Buildpacks are being used is building production images, but this is only one piece of the software development process. Without a solid buildpack test environment story, users will be required use alternatives for building their environment. A testing environment, while different, shares many of the same broad strokes ultimately producing an execution environment. This is something Buildpacks are well positioned to solve. # What it is [what-it-is]: #what-it-is @@ -70,11 +70,11 @@ The specifics of creating development enviroments are out of scope of this RFC, ## `exec-env` key in TOML -In order to support additional execution environments an `exec-env` key will be added to various TOML tables in the project. The value can be any string with `all` having special meaning. `all` will apply to all execution environments and will be the default if not specified. This should make it backwards compatible and optional. When `exec-env` is not set to `all`, the table settings will only be applied to that execution environment. +In order to support additional execution environments an `exec-env` key will be added to various TOML tables in the project. This will be an array that takes string values. An individual element can be any string with `*` having special meaning. Similar to the ["any stack RFC"](https://github.com/buildpacks/rfcs/blob/main/text/0056-any-stack-buildpacks.md) `*` will apply to all execution environments. `["*"]` will be the default if not specified. This should make the key backwards compatible and optional. When `exec-env` is not empty and does not include `*`, the table settings will only be applied to the specificed execution environments. ### Project Descriptor - `project.toml` (App Developers) -This file be extended by adding `exec-env` to the following tables: +An app developer may have execution environment configuration like only using a metrics agent in production or headless user agent needs in test. In order to facilitate these needs, the project descriptor will be extended by adding `exec-env` to the following tables: `[[io.buildpacks.group]]` `[[io.buildpacks.pre.group]]` @@ -91,15 +91,20 @@ schema-version = "0.3" id = "buildpacks/ruby" version = "latest" +[[io.buildpacks.group]] +id = "buildpacks/nodejs" +version = "latest" +exec-env = ["production", "test"] + [[io.buildpacks.group]] id = "buildpacks/metrics-agent" version = "latest" -exec-env = "production" +exec-env = ["production"] [[io.buildpacks.group]] id = "buildpacks/headless-chrome" version = "latest" -exec-env = "test" +exec-env = ["test"] [[io.buildpacks.post.group]] id = "buildpacks/procfile" @@ -108,30 +113,38 @@ version = "latest" [[io.buildpacks.build.env]] name = "RAILS_ENV" value = "production" -exec-env = "production" +exec-env = ["production"] [[io.buildpacks.build.env]] name = "RAILS_ENV" value = "test" -exec-env = "test" +exec-env = ["test"] [[io.buildpacks.build.env]] name = "PARALLEL_WORKERS" value = "4" -exec-env = "test" +exec-env = ["production"] ``` ### `builder.toml` (Builder Authors) -The only table that `exec-env` will be added to is `[[order.group.env]]`. +The the only table `exec-env` will be added to is `[[order.group]]` and `[[build.env]]`. ### `buildpack.toml` (Buildpack Authors) The only table that `exec-env` will be added to is `[[buildpack.order.group]]`. This only is applicable for composite buildpacks. +### `launch.toml` (Buildpack Authors) + +Not all process types make sense for every execution environment. In order to help hint to the platform the intention how a process should be used, the `exec-env` key will be added to the `[[processes]]` table. + +### `metadata.toml` (Platform Operators) + +On the platform side, the `exec-env` key will be added to `metadata.toml` in the `[[processes]]` table to mirror `launch.toml`. This will help platforms to make decisions on which processes are desired for each execution environment. `lifecycle` will list all processes and will not exclude any based on the execution environment Instead, a platform can use the available information to make a decision. + ## `CNB_EXEC_ENV` Environment Variable -This env var will reserve the following values: +The spec will reserve the following values to help standardize execution environments: * production * test @@ -139,7 +152,9 @@ This env var will reserve the following values: ### Buildpack API -A buildpack author will be able to determine the execution environment their buildpack is expected to build for by reading the `CNB_EXEC_ENV` environment variable. If this value is not set, a Buildpack Author can assume it's set to `production`. +A buildpack author will be able to determine the execution environment their buildpack is expected to build for by reading the `CNB_EXEC_ENV` environment variable. If this value is not set, a Buildpack Author can assume it's set to `production`. This will be provided for both `bin/detect` and `bin/build`. + +This would let a buildpack author do different things based on the execution environment. For example, it's common for a production build to not include test dependencies or files, while a test environment would include those things. For compiled languages, a production might even remove the source code and just leave the compiled binary with optimizations. In a test environment, it may include debug symbols, not run with optimizations, not prune the source tree. ### Platform API @@ -197,6 +212,8 @@ The original Cloud Native Buildpacks spec included a Develop API, but it was nev - strings can help account for use cases we haven't thought of yet. - Should buildpacks be allowed specify allowlist execution environments? - What changes are needed in the buildpack registry? +- Does `build.env` need to support execution environments in `builder.toml`? +- Should the reserved exec env strings be namespaced? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 7f3ac0f0ccd039af33058be90a9162a771bd548a Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Tue, 7 Jan 2025 10:14:48 -0600 Subject: [PATCH 208/222] add label Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 837c0243d..564ced647 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -76,10 +76,10 @@ In order to support additional execution environments an `exec-env` key will be An app developer may have execution environment configuration like only using a metrics agent in production or headless user agent needs in test. In order to facilitate these needs, the project descriptor will be extended by adding `exec-env` to the following tables: -`[[io.buildpacks.group]]` -`[[io.buildpacks.pre.group]]` -`[[io.buildpacks.post.group.env]]` -`[[io.buildpacks.build.env]]` +* `[[io.buildpacks.group]]` +* `[[io.buildpacks.pre.group]]` +* `[[io.buildpacks.post.group.env]]` +* `[[io.buildpacks.build.env]]` An example would look like this: @@ -160,6 +160,8 @@ This would let a buildpack author do different things based on the execution env It will be up to the platform to set the environment variable `CNB_EXEC_ENV`. If this value is set, `lifecycle` MUST NOT override this value. If the value is not set, `lifecycle` will set it to `production`. +During the export phase, `lifecycle` will set the `io.buildpacks.exec-env` label with the value from the `CNB_EXEC_ENV` environment variable. This will make it easier for anyone examining the OCI image to determine the execution environment. + # Migration [migration]: #migration @@ -214,6 +216,7 @@ The original Cloud Native Buildpacks spec included a Develop API, but it was nev - What changes are needed in the buildpack registry? - Does `build.env` need to support execution environments in `builder.toml`? - Should the reserved exec env strings be namespaced? +- Instead of creating a new label, should we stash it into a JSON label? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 90f6695ca752438a9a10c317e824e0abdb52a56a Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 16 Jan 2025 18:43:06 -0600 Subject: [PATCH 209/222] updates from WG feedback Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 564ced647..40fd47af0 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -132,7 +132,9 @@ The the only table `exec-env` will be added to is `[[order.group]]` and `[[build ### `buildpack.toml` (Buildpack Authors) -The only table that `exec-env` will be added to is `[[buildpack.order.group]]`. This only is applicable for composite buildpacks. +As a piece of metadata, a buildpack should list the execution environments it supports in the `[[buildpack.exec-env]]` array. This can be used by the buildpack registry in the future. + +For composite buildpacks, `exec-env` will be added to is `[[buildpack.order.group]]`. ### `launch.toml` (Buildpack Authors) @@ -144,12 +146,18 @@ On the platform side, the `exec-env` key will be added to `metadata.toml` in the ## `CNB_EXEC_ENV` Environment Variable +This value is a string and must abide by similar rules we use for IDs: + +* MUST only contain numbers, letters, and the characters `.`, and `-`. + The spec will reserve the following values to help standardize execution environments: * production * test * development +In addition, the `/` character is reserved in case we need to introduce namespacing in the future. + ### Buildpack API A buildpack author will be able to determine the execution environment their buildpack is expected to build for by reading the `CNB_EXEC_ENV` environment variable. If this value is not set, a Buildpack Author can assume it's set to `production`. This will be provided for both `bin/detect` and `bin/build`. @@ -208,15 +216,22 @@ The original Cloud Native Buildpacks spec included a Develop API, but it was nev [unresolved-questions]: #unresolved-questions - "env" is overloaded as a word since we also use it for environment variables. Is there a better word here? +While "env" is overloaded, it matches the intent. - Should there be builders that are specific to an execution environment? What about `include` or `exclude`? +No reason right now to restrict builders to an execution environment. - Should the execution environments be an enum or flexible as a string? - enums will help encourage standardization across buildpacks and platforms. - strings can help account for use cases we haven't thought of yet. +We're going to opt for a string to be flexible. - Should buildpacks be allowed specify allowlist execution environments? +We're currently optimizing for app developer flexibility to use the buildpacks as they see fit. - What changes are needed in the buildpack registry? +The buildpack registry will need to expand the index to support the `buildpack.toml`'s `exec-env` field. - Does `build.env` need to support execution environments in `builder.toml`? - Should the reserved exec env strings be namespaced? +We're going to reserve the `/` character for namespacing if needed in the future. - Instead of creating a new label, should we stash it into a JSON label? +The current JSON blobs are exports of lifecycle files, so we're going to create a new one for now. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 70a887379c7429ff65707417ffa5954c9fb6666c Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 6 Mar 2025 10:07:55 -0800 Subject: [PATCH 210/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 40fd47af0..271606363 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -12,7 +12,7 @@ # Summary [summary]: #summary -Add support for different execution environments for buildpacks, with testing as the first use case. +Add buildpack support for multiple execution environments, with better support for testing as the first use case. # Definitions [definitions]: #definitions From 5e4554387fcfb328d8606bcc51d9afb59d90d676 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 10:59:46 -0500 Subject: [PATCH 211/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 271606363..52f726ecf 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -17,7 +17,7 @@ Add buildpack support for multiple execution environments, with better support f # Definitions [definitions]: #definitions -- Execution Environment - The target environment the OCI image is expected to be run in, i.e. production, test, development. +- Execution Environment - The target environment the OCI image is expected to be executed in, i.e. production, test, development. # Motivation [motivation]: #motivation From 6c046651d9e976ae66f9b5fab429ef66c210a7e1 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:00:04 -0500 Subject: [PATCH 212/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 52f726ecf..54c4f54a3 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -22,7 +22,7 @@ Add buildpack support for multiple execution environments, with better support f # Motivation [motivation]: #motivation -The main way Buildpacks are being used is building production images, but this is only one piece of the software development process. Without a solid buildpack test environment story, users will be required use alternatives for building their environment. A testing environment, while different, shares many of the same broad strokes ultimately producing an execution environment. This is something Buildpacks are well positioned to solve. +Buildpacks are mainly being used to build images for production environments, but this is only one piece of the software development process. Without test environment support for buildpacks, users currently have to implement alternatives for testing their source code. Buildpacks are well positioned to help bridge that gap, as building production and test environments often share many of the same broad strokes. # What it is [what-it-is]: #what-it-is From 7d4e11b780e830b3edb8127b929d478fe92c0ef9 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:03:53 -0500 Subject: [PATCH 213/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 54c4f54a3..128dcd3ba 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -58,7 +58,7 @@ This narrows the scope of the Buildpack responsibilities to what it's already go - launch.toml with test process marked as default (recommendation) ## Setting the Execution Enviroment for Build + Buildpack -A platform will set the `CNB_EXEC_ENV` env var to the execution environment desired. Buildpacks can than read this env var to branch or switch on logic needed based on the execution environment. +A platform will set the `CNB_EXEC_ENV` env var to the desired execution environment. Buildpacks can then read this env var to branch or switch on logic needed for the execution environment. In addition, Builder Authors, Buildpack Authors, and App Developers will be able to specify various options to a specific execution enviroment using the `exec-env` key. From 1051e3abab9de2ff3d3155aaf68cdd0b8a96e286 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:29:09 -0500 Subject: [PATCH 214/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 128dcd3ba..81f64a26e 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -60,7 +60,7 @@ This narrows the scope of the Buildpack responsibilities to what it's already go ## Setting the Execution Enviroment for Build + Buildpack A platform will set the `CNB_EXEC_ENV` env var to the desired execution environment. Buildpacks can then read this env var to branch or switch on logic needed for the execution environment. -In addition, Builder Authors, Buildpack Authors, and App Developers will be able to specify various options to a specific execution enviroment using the `exec-env` key. +In addition, Builder Authors, Buildpack Authors, and App Developers will be able to configure various options for specific execution environments using the `exec-env` key. ## Development Environments The specifics of creating development enviroments are out of scope of this RFC, but it's not hard to extrapolate how these kind of changes can assist in creating Buildpacks for development environments. From 80cccea9959b130de30872ebc58c26bfaaebf87f Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:30:02 -0500 Subject: [PATCH 215/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 81f64a26e..711919041 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -74,7 +74,7 @@ In order to support additional execution environments an `exec-env` key will be ### Project Descriptor - `project.toml` (App Developers) -An app developer may have execution environment configuration like only using a metrics agent in production or headless user agent needs in test. In order to facilitate these needs, the project descriptor will be extended by adding `exec-env` to the following tables: +An app developer may have execution environment specific configuration, like only using a metrics agent in production, or a headless user agent in test. In order to facilitate these needs, the project descriptor will be extended by adding `exec-env` to the following tables: * `[[io.buildpacks.group]]` * `[[io.buildpacks.pre.group]]` From 495ad996ace720432fb1121c1556a7691791fdf3 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:30:29 -0500 Subject: [PATCH 216/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 711919041..0872e7690 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -128,7 +128,7 @@ exec-env = ["production"] ### `builder.toml` (Builder Authors) -The the only table `exec-env` will be added to is `[[order.group]]` and `[[build.env]]`. +The `exec-env` key will be added to the `[[order.group]]` and `[[build.env]]` arrays of tables. ### `buildpack.toml` (Buildpack Authors) From a5091927cb96202701e220e57d8ff6f26601c230 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:35:32 -0500 Subject: [PATCH 217/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 0872e7690..01aa03a77 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -162,7 +162,7 @@ In addition, the `/` character is reserved in case we need to introduce namespac A buildpack author will be able to determine the execution environment their buildpack is expected to build for by reading the `CNB_EXEC_ENV` environment variable. If this value is not set, a Buildpack Author can assume it's set to `production`. This will be provided for both `bin/detect` and `bin/build`. -This would let a buildpack author do different things based on the execution environment. For example, it's common for a production build to not include test dependencies or files, while a test environment would include those things. For compiled languages, a production might even remove the source code and just leave the compiled binary with optimizations. In a test environment, it may include debug symbols, not run with optimizations, not prune the source tree. +This would let a buildpack author do different things based on the execution environment. For example, it's common for a production build to exclude test dependencies or files that a test environment would include. For compiled languages, a production build might even remove the source code and just leave the compiled binary with optimizations. In a test environment, a build may include debug symbols, be compiled without optimizations, and leave the source code untouched. ### Platform API From 1aadb3c5cd149859f1848142829a37806504d1ac Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:36:08 -0500 Subject: [PATCH 218/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 01aa03a77..8dcef2203 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -200,7 +200,7 @@ With Project Descriptor, this would remove the need for the `exec-env` key at th ## [Heroku Testpack API](https://devcenter.heroku.com/articles/testpack-api) -As part of the classic buildpack design, Heroku extended the API to include support for tests. It added a `bin/test-compile` phase, which is like the normal `bin/compile` phase but specific for setting things up for test. `bin/test` was called for executing tests. This design let a buildpack have a specific codepath for tests and also do intelligent logic for executing tests based on the source code. For example [in Ruby](https://github.com/heroku/heroku-buildpack-ruby/blob/main/bin/support/ruby_test#L62-L76), the buildpack author can run `bin/rspec` if it detects `rspec`, `bin/rails test` when using rails, or default to `rake test`. +As part of the classic buildpack design, Heroku extended the API to include support for tests. It added a `bin/test-compile` phase, which is similar to the `bin/compile` phase, but specific for setting things up for test. `bin/test` was called for executing tests. This design lets buildpack authors write codepaths and logic optimized specifically for executing tests based on the source code. For example, [this Ruby buildpack](https://github.com/heroku/heroku-buildpack-ruby/blob/fb393cb46e23ab809e21daeef2a97cbd5f04a370/bin/support/ruby_test#L62-L76) will run `bin/rspec` if it detects `rspec`, `bin/rails test` when using rails, or default to `rake test`. There were some flaws in this design. Though it's clean to separate production and test code paths, they end up sharing a lot of code. Many of the bash based Heroku buildpacks would just [call `bin/compile`](https://github.com/heroku/heroku-buildpack-nodejs/blob/main/bin/test-compile#L24) with different parameters/env vars. From d38b2c0f797d5fb2ddfa6bd81b2a02fcbf4702e3 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:36:28 -0500 Subject: [PATCH 219/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 8dcef2203..4d9261f48 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -142,7 +142,7 @@ Not all process types make sense for every execution environment. In order to he ### `metadata.toml` (Platform Operators) -On the platform side, the `exec-env` key will be added to `metadata.toml` in the `[[processes]]` table to mirror `launch.toml`. This will help platforms to make decisions on which processes are desired for each execution environment. `lifecycle` will list all processes and will not exclude any based on the execution environment Instead, a platform can use the available information to make a decision. +On the platform side, the `exec-env` key will be added to `metadata.toml` in the `[[processes]]` table to mirror `launch.toml`. This will help platforms decide which processes are desired for each execution environment. `lifecycle` will list all processes and will not exclude any based on the execution environment Instead, a platform can use the available information to make a decision. ## `CNB_EXEC_ENV` Environment Variable From ca65826189a179ed698ec19f06f2dcd25899da46 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:37:23 -0500 Subject: [PATCH 220/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 4d9261f48..1a8139389 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -63,7 +63,7 @@ A platform will set the `CNB_EXEC_ENV` env var to the desired execution environm In addition, Builder Authors, Buildpack Authors, and App Developers will be able to configure various options for specific execution environments using the `exec-env` key. ## Development Environments -The specifics of creating development enviroments are out of scope of this RFC, but it's not hard to extrapolate how these kind of changes can assist in creating Buildpacks for development environments. +The specifics of creating development environments are out of scope of this RFC, but it's not hard to extrapolate how the proposed changes can assist in creating Buildpacks for development environments. # How it Works [how-it-works]: #how-it-works From ba29b350bd9827fd743a89cf6f30280161102695 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:37:48 -0500 Subject: [PATCH 221/222] Update text/0000-execution-environments.md Co-authored-by: Rune Soerensen Signed-off-by: Terence Lee --- text/0000-execution-environments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-execution-environments.md b/text/0000-execution-environments.md index 1a8139389..4e0737769 100644 --- a/text/0000-execution-environments.md +++ b/text/0000-execution-environments.md @@ -70,7 +70,7 @@ The specifics of creating development environments are out of scope of this RFC, ## `exec-env` key in TOML -In order to support additional execution environments an `exec-env` key will be added to various TOML tables in the project. This will be an array that takes string values. An individual element can be any string with `*` having special meaning. Similar to the ["any stack RFC"](https://github.com/buildpacks/rfcs/blob/main/text/0056-any-stack-buildpacks.md) `*` will apply to all execution environments. `["*"]` will be the default if not specified. This should make the key backwards compatible and optional. When `exec-env` is not empty and does not include `*`, the table settings will only be applied to the specificed execution environments. +In order to support additional execution environments, an optional `exec-env` key will be added to various TOML tables in the project. This will be an array that takes string values. An individual element can be any string with `*` having special meaning. Similar to the ["any stack RFC"](https://github.com/buildpacks/rfcs/blob/main/text/0056-any-stack-buildpacks.md), `*` will apply to all execution environments. `["*"]` will be the default if not specified. This should make the key backwards compatible. When `exec-env` is not empty and does not include `*`, the table settings will only be applied to the specified execution environments. ### Project Descriptor - `project.toml` (App Developers) From 6ce039b680dc7b4d2ab5f2fa50ef6523e9b3f7f1 Mon Sep 17 00:00:00 2001 From: Terence Lee Date: Thu, 20 Mar 2025 11:51:05 -0500 Subject: [PATCH 222/222] RFC 0134 Signed-off-by: Terence Lee --- ...ion-environments.md => 0134-execution-environments.md} | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) rename text/{0000-execution-environments.md => 0134-execution-environments.md} (98%) diff --git a/text/0000-execution-environments.md b/text/0134-execution-environments.md similarity index 98% rename from text/0000-execution-environments.md rename to text/0134-execution-environments.md index 4e0737769..cf10b2491 100644 --- a/text/0000-execution-environments.md +++ b/text/0134-execution-environments.md @@ -3,11 +3,11 @@ - Name: Execution Environments - Start Date: 2023-01-11 - Author(s): @hone -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: [rfcs#274](https://github.com/buildpacks/rfcs/pull/274) - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) -- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) +- CNB Issue: [rfcs#327](https://github.com/buildpacks/rfcs/issues/327) +- Supersedes: N/A # Summary [summary]: #summary