Technical-proposal needed for compliance and certification

[vbatts]

Ref #452 (comment)

The changes for validation need to have a bit of guidance as to the nature of the compliance validation.
This would ideally be a product from the cert and compliance working group to ensure validation of OCI image artifacts satisfies the requirements.

[vbatts]

cc @opencontainers/certification-maintainers

[wking]

On Wed, Feb 15, 2017 at 02:40:57PM -0800, Vincent Batts wrote: This would ideally be a product from the cert and compliance working group to ensure validation of OCI image artifacts satisfies the requirements.

In opencontainers/image-tools#66, I floated “strict” (only contains OCI-defined content) and “base” (only contains content which the image-spec REQUIREs unpackers to support) levels for compliance testing. Both of those are more restrictive than a “legal” level (content does not violate any MUST-level image-spec requirements). Another axis which cert/compliance would be interested in is MUST-level vs. SHOULD-level validation. So I think we should make it easy for callers to validate with any combination of legal/strict/base and legal/should. Beyond the cert/compliance working group, I also floated an “unpackable” level (can image-tools unpack the content). That would not impact the cert/compliance working group, but it would be unfortunate to address only legal/strict/base (or whatever) in image-spec and leave no way for image-tools to support legal/strict/unpackable/base. It would also be nice if callers could register support for additional media types (e.g. “I can also unpack application/vnd.example.layer.v3.json”). And it should be easy for Go callers to figure out about validator concerns (e.g. “what were the SHOULD-level violations, if any?”) which is hard if the validation code is printing warnings to stdout [1]. I've taken a few stabs at implementating a framework for this with no success (e.g. in #403), so I'll leave figuring that out to runtime-tools maintainers. The above stuff is my take on the constraints for that framework. This issue may just be about collecting constraints anyway. [1]: #491 (comment)

[RobDolinMS]

@vbatts Would the proposal at: opencontainers/oci-conformance#26 fulfill this issue?

[vbatts]

@RobDolinMS perhaps so

[vbatts]

The conformance issue was closed because it went stale. I still think a conformance/validation piece would be nice here.

[tianon]

I think applying programmatic conformance tests to the image-spec is a tricky proposition -- what would you verify? That a given OCI layout is conformant? That a given object in a registry matches the text? How would you verify that a given "implementation" follows the spec, especially in a generic way, and especially when they all do such very different things?

(It's not like the distribution-spec, where there's a clear API that can ~easily be validated to be conformant to a large portion of the spec, but even there we have similar struggles with making sure clients are conformant because client use cases are so widely varied.)

Given this complexity, I would suggest that we should probably close this (and open a new issue if someone comes up with a clever way to overcome the obstacles to verifying something meaningful in implementations here).

[sudo-bmitch]

I'm not particularly attached to this PR, but adding an image-spec conformance test is in my longer term to-do list. I can see two types of conformance based on the OCI Layout. One test would consume a provided Layout to verify all of the contents match the spec requirements. And the second test could be an example OCI Layout containing various images that tools should be able to consume (or at least not error if they encounter).

[tianon]

Heh, what you've described is very similar to #637 (IMO we should close one or the other).

[sudo-bmitch]

It describes the second (example input for consumers), but not the first (verifying output from producers). I'm not picky on whether that's two issues or one. Even without an issue, it's on my own to-do list to get to one day.