Support continuous model validation #89

miyunari · 2026-01-25T18:39:03Z

Provide optional continuous model validation supported on k8s 1.28 or newer.

The continuousValidation entry currently only supports checking the model
on a given interval.
In the future we may want to extend it to support things like:

onCRDChange
onSigChange
...

Example:

apiVersion: ml.sigstore.dev/v1alpha1
kind: ModelValidation
metadata:
  name: continuous-signed-test
spec:
  config:
    publicKeyConfig:
      keyPath: /keys/test_public_key.pub
  model:
    path: /data
    signaturePath: /data/model.sig
  imagePullPolicy: Always
  continuousValidation:
    enabled: true
    interval: "5m"

Signed-off-by: Nina Bongartz pnink@web.de

miyunari · 2026-02-03T11:20:53Z

cmd/validation-agent/main.go

+}
+
+func runValidation(args []string, logger logr.Logger) error {
+	cmd := exec.Command("/usr/local/bin/model_signing", args...)


In the future it might be more ideal to use the go library.

Provide optional continuous model validation supported on k8s 1.28 or newer. The continuousValidation entry currently only supports checking the model on a given interval. In the future we may want to extend it to support things like: - onCRDChange - onSigChange - ... Example: ```yaml apiVersion: ml.sigstore.dev/v1alpha1 kind: ModelValidation metadata: name: continuous-signed-test spec: config: publicKeyConfig: keyPath: /keys/test_public_key.pub model: path: /data signaturePath: /data/model.sig imagePullPolicy: Always continuousValidation: enabled: true interval: "5m" ``` Signed-off-by: Nina Bongartz <pnink@web.de>

E2E tests were failing with Init:ErrImagePull because the operator expected ghcr.io/sigstore/model-validation-agent:v0.1.0 but e2e built ghcr.io/sigstore/model-validation-operator-agent:v0.0.1. Fix by building the agent image with the correct name that matches the hardcoded constant in internal/constants/images.go. Also set imagePullPolicy: IfNotPresent in test templates to ensure e2e tests use locally loaded images instead of trying to pull from the registry. Signed-off-by: Nina Bongartz <pnink@web.de>

The e2e test "should successfully validate with public key signature" was failing with permission denied errors when the validation-agent tried to read model files. Root cause: - The validation-agent runs as UID 65532 (non-root) per Dockerfile.agent - The model-data-setup DaemonSet runs as root and copies files to hostPath - Files copied by root were not readable by the non-root agent process Fix: - Added chmod -R a+rX to make files readable by all users - The 'a+rX' flag makes files readable and directories executable - Capital X only adds execute on directories, not regular files This allows the validation-agent to successfully read model files during validation. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Nina Bongartz <pnink@web.de>

SequeI · 2026-02-09T11:07:22Z

Makefile


+# AGENT_IMG defines the image:tag used for the validation agent.
+# Use the same agent image name as hardcoded in internal/constants/images.go
+AGENT_IMG ?= ghcr.io/sigstore/model-validation-agent:v0.1.0


As mentioned in comment, we probably shouldn't hardcode the same image version in two different places, might cause image drift in the future/not needed extra maintenance

It can be added as a build flag

SequeI · 2026-02-09T11:15:24Z

cmd/validation-agent/main.go

+
+	if err := server.ListenAndServe(); err != nil {
+		logger.Error(err, "Health server failed")
+		os.Exit(1)


We should have some sort of graceful shutdown for the health server, otherwise it will keep running until terminated if we have an issue with the main app

+1, we should be able to pass in a context and have that trigger the shutdown.

SequeI · 2026-02-09T11:18:57Z

api/v1alpha1/modelvalidation_types.go

+	// Interval defines how often to re-validate the model (e.g., "5m", "1h").
+	// Only used when Enabled is true.
+	// +kubebuilder:default="5m"
+	// +kubebuilder:validation:Pattern=`^([0-9]+(\.[0-9]+)?(s|m|h))+$`


This regex allows for patterns like 1s and such, we probably don't want to cause massive CPU usage. We should add a minimum interval like 1m/5m to prevent a DOS scenario

SequeI · 2026-02-09T11:19:55Z

internal/webhooks/pod_webhook.go

+
+	container := corev1.Container{
 		Name:            constants.ModelValidationInitContainerName,
+		Image:           constants.ModelValidationAgentImage,


From my understanding, even if continuous validation is not enabled, we still use ModelValidationAgentImage instead of ModelTransparencyCliImage?

SequeI · 2026-02-09T11:21:16Z

internal/webhooks/pod_webhook.go

+		imagePullPolicy = mv.Spec.ImagePullPolicy
+	}
+
+	container := corev1.Container{


We should add resource limits to the sidecar container to prevent unbounded resource allocation

SequeI · 2026-02-09T11:22:42Z

internal/webhooks/pod_webhook.go

+		if pp.Annotations == nil {
+			pp.Annotations = make(map[string]string)
+		}
+		pp.Annotations[constants.ContinuousValidationAnnotationKey] = "true"


I don't see this annotation actually being used anywhere, is this purely just for tracking/not to be used programatically to achieve something?

SequeI · 2026-02-09T11:28:36Z

Looks good so far, left some questions/review on the PR :)

knrc · 2026-02-10T21:05:03Z

api/v1alpha1/modelvalidation_types.go

 }

+// ContinuousValidation defines the configuration for continuous model validation.
+// When enabled, the validation container runs as a native sidecar (requires Kubernetes 1.28+)


FYI this feature is only enabled by default from 1.29, when it became beta, and needs to be enabled in 1.28

knrc · 2026-02-10T21:13:37Z

Makefile


+# AGENT_IMG defines the image:tag used for the validation agent.
+# Use the same agent image name as hardcoded in internal/constants/images.go
+AGENT_IMG ?= ghcr.io/sigstore/model-validation-agent:v0.1.0


It can be added as a build flag

knrc · 2026-02-10T21:25:55Z

cmd/validation-agent/main.go

+
+	if err := server.ListenAndServe(); err != nil {
+		logger.Error(err, "Health server failed")
+		os.Exit(1)


+1, we should be able to pass in a context and have that trigger the shutdown.

miyunari force-pushed the continues_check branch from 49f515e to 0da7550 Compare February 3, 2026 11:17

miyunari changed the title ~~poc: continuous model validation~~ Support continuous model validation Feb 3, 2026

miyunari force-pushed the continues_check branch from 0da7550 to 71e1468 Compare February 3, 2026 11:19

miyunari commented Feb 3, 2026

View reviewed changes

miyunari force-pushed the continues_check branch 4 times, most recently from de401ee to fb05249 Compare February 7, 2026 12:08

Nina Bongartz added 2 commits February 7, 2026 13:12

miyunari force-pushed the continues_check branch 2 times, most recently from 30ee8ab to a3a9e13 Compare February 7, 2026 12:45

miyunari force-pushed the continues_check branch from a3a9e13 to c9b93d8 Compare February 7, 2026 12:59

miyunari requested review from SequeI and knrc February 7, 2026 13:00

miyunari marked this pull request as ready for review February 7, 2026 13:00

SequeI requested changes Feb 9, 2026

View reviewed changes

knrc approved these changes Feb 10, 2026

View reviewed changes

sampras343 mentioned this pull request Feb 12, 2026

Donating model-transparency-go sigstore/TSC#61

Open

Support continuous model validation #89

Are you sure you want to change the base?

Support continuous model validation #89

Uh oh!

Conversation

miyunari commented Jan 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SequeI commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

miyunari commented Jan 25, 2026 •

edited

Loading

SequeI commented Feb 9, 2026 •

edited

Loading