Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions deploy/crd/kcp.io/syncagent.kcp.io_publishedresources.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -778,6 +778,35 @@ spec:
Version is the API version of the related resource. This can be left blank to automatically
use the preferred version.
type: string
watch:
description: |-
Watch configures how the agent identifies the owning primary object when a related
resource with origin: kcp changes. When set, the agent sets up a watch on the related
resource type and uses the configured rule to enqueue the correct primary object.
Without this field, changes to origin:kcp related resources do not trigger reconciliation.
properties:
byLabel:
additionalProperties:
type: string
description: |-
ByLabel configures the watch handler to list primary objects matching a label selector
derived from the changed object. Each map key is a label key on the primary object;
each value is a Go template expression evaluated with the changed object available as
.watchObject (with fields .name, .namespace, .labels).
type: object
byOwner:
description: |-
ByOwner configures the watch handler to inspect the OwnerReferences of the changed
object. When an OwnerReference with the given Kind is found, the referenced owner
is enqueued as the primary object.
properties:
kind:
description: Kind is the Kind to look for in the OwnerReferences of the changed related object.
type: string
required:
- kind
type: object
type: object
required:
- identifier
- object
Expand Down
2 changes: 2 additions & 0 deletions hack/tools.checksums
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
boilerplate|GOARCH=amd64;GOOS=linux|6f05fc3be207ae2ed99e125509a08df677cb007e197e16607c654a434b91d47f
boilerplate|GOARCH=arm64;GOOS=darwin|3ac82c58f440ac8461746674e39311ba332d6d960966a060dd3be734b1111522
boilerplate|GOARCH=arm64;GOOS=linux|70253486ed7a803a35a9abb2bab4db2f1f7748d5266bf7a1c2ee298fda2b208a
etcd|GOARCH=amd64;GOOS=linux|435d74510f3216bab1932fb6d7a6b5fe8245301143fcd25f7e65dfb7dcf8904a
etcd|GOARCH=arm64;GOOS=linux|cc8c645e5a8df0f35f2a5c51d9b9383037eef0cf0167c52e648457b3971a7a09
Expand All @@ -11,4 +12,5 @@ kube-apiserver|GOARCH=arm64;GOOS=linux|6ade6c2646e2c01fde1095407452afc2b65e89d6d
kubectl|GOARCH=amd64;GOOS=linux|9591f3d75e1581f3f7392e6ad119aab2f28ae7d6c6e083dc5d22469667f27253
kubectl|GOARCH=arm64;GOOS=linux|95df604e914941f3172a93fa8feeb1a1a50f4011dfbe0c01e01b660afc8f9b85
yq|GOARCH=amd64;GOOS=linux|0c2b24e645b57d8e7c0566d18643a6d4f5580feeea3878127354a46f2a1e4598
yq|GOARCH=arm64;GOOS=darwin|164e10e5f7df62990e4f3823205e7ea42ba5660523a428df07c7386c0b62e3d9
yq|GOARCH=arm64;GOOS=linux|9477ac3cc447b6c083986129e35af8122eb2b938fe55c9c3e40436fb966e5813
70 changes: 70 additions & 0 deletions internal/controller/sync/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,12 @@ import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/utils/ptr"
ctrlruntimeclient "sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/cluster"
"sigs.k8s.io/controller-runtime/pkg/handler"
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/predicate"
Expand Down Expand Up @@ -161,6 +164,73 @@ func Create(
return nil, fmt.Errorf("failed to setup local-side watch: %w", err)
}

// Watch origin:kcp related resources so that changes to them trigger reconciliation
// of the owning primary object. Only related resources with a Watch config are covered.
watchedGVKs := sets.New[schema.GroupVersionKind]()
for _, relRes := range pubRes.Spec.Related {
if relRes.Origin != syncagentv1alpha1.RelatedResourceOriginKcp || relRes.Watch == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't quite remember why we would only want to do this on the kcp side.. Couldn't we technically also have related objects on the service cluster side? It feels like this watching behaviour should work regardless of the origin side, no? 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I.e. it should always watch on the origin side, regardless where that side actually is.

continue
}

gvr := schema.GroupVersionResource{
Group: relRes.Group,
Version: relRes.Version,
Resource: relRes.Resource,
}

// Use the local REST mapper to determine the Kind.
gvk, err := localManager.GetRESTMapper().KindFor(gvr)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the related resource originates in kcp (as per the first if statement in this loop), then why are we using the local (= service cluster) REST mapper to resolve it?

Related resources support projection (i.e. changing their GVK when syncing from one side to another), so a GVK that exists in kcp is not necessarily the same as it is on the service cluster.

Ideally this should use a restmapper of the origin side (whereever that might be).

if err != nil {
log.Warnw("Failed to determine Kind for origin:kcp related resource, skipping watch", "gvr", gvr, "error", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fear this log message will never be seen by anyone and during runtime, this state will also not fix itself (the agent won't try to re-establish this watch at a later time, unless you restart the entire agent).

To catch misconfigurations, wouldn't it make more sense to error out here?

continue
}

// Deduplicate: only set up one watch per GVK.
if watchedGVKs.Has(gvk) {
continue
}
watchedGVKs.Insert(gvk)

relatedDummy := &unstructured.Unstructured{}
relatedDummy.SetGroupVersionKind(gvk)

var enqueueForRelated mchandler.TypedEventHandlerFunc[*unstructured.Unstructured, mcreconcile.Request]

switch {
case relRes.Watch.ByOwner != nil:
ownerKind := relRes.Watch.ByOwner.Kind
enqueueForRelated = func(clusterName string, _ cluster.Cluster) handler.TypedEventHandler[*unstructured.Unstructured, mcreconcile.Request] {
return &byOwnerEventHandler{
clusterName: clusterName,
ownerKind: ownerKind,
}
}

case relRes.Watch.ByLabel != nil:
labelTemplates := relRes.Watch.ByLabel
primaryDummy := remoteDummy.DeepCopy()
enqueueForRelated = func(clusterName string, cl cluster.Cluster) handler.TypedEventHandler[*unstructured.Unstructured, mcreconcile.Request] {
return &byLabelEventHandler{
clusterName: clusterName,
client: cl.GetClient(),
primaryDummy: primaryDummy,
labelTemplates: labelTemplates,
log: log,
}
}

default:
log.Warnw("origin:kcp related resource has Watch set but neither byOwner nor byLabel configured, skipping", "gvk", gvk)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should definitely be an error. Misconfigured PublishedResources should IMHO scream loudly. If we have the XValidation rule such errors will be much less likely, but I would still error out if the PublishedResource is broken.

continue
}

if err := c.MultiClusterWatch(mcsource.TypedKind(relatedDummy, enqueueForRelated)); err != nil {
return nil, fmt.Errorf("failed to setup watch for origin:kcp related resource %v: %w", gvk, err)
}

log.Infow("Set up watch for origin:kcp related resource", "gvk", gvk)
}

log.Info("Done setting up unmanaged controller.")

return c, nil
Expand Down
144 changes: 144 additions & 0 deletions internal/controller/sync/related_handlers.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
/*
Copyright 2026 The KCP Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package sync

import (
"context"

"go.uber.org/zap"

"github.com/kcp-dev/api-syncagent/internal/sync/templating"

"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/util/workqueue"
ctrlruntimeclient "sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/event"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
mcreconcile "sigs.k8s.io/multicluster-runtime/pkg/reconcile"
)

// byOwnerEventHandler enqueues the primary object by inspecting the OwnerReferences
// of the changed related object and finding one with the configured Kind.
type byOwnerEventHandler struct {
clusterName string
ownerKind string
}

func (h *byOwnerEventHandler) Create(_ context.Context, evt event.TypedCreateEvent[*unstructured.Unstructured], q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
h.enqueue(evt.Object, q)
}

func (h *byOwnerEventHandler) Update(_ context.Context, evt event.TypedUpdateEvent[*unstructured.Unstructured], q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
h.enqueue(evt.ObjectNew, q)
}

func (h *byOwnerEventHandler) Delete(_ context.Context, evt event.TypedDeleteEvent[*unstructured.Unstructured], q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
h.enqueue(evt.Object, q)
}

func (h *byOwnerEventHandler) Generic(_ context.Context, evt event.TypedGenericEvent[*unstructured.Unstructured], q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
h.enqueue(evt.Object, q)
}

func (h *byOwnerEventHandler) enqueue(obj *unstructured.Unstructured, q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
for _, ref := range obj.GetOwnerReferences() {
if ref.Kind == h.ownerKind {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should compare G, V and K, I think.

q.Add(mcreconcile.Request{
ClusterName: h.clusterName,
Request: reconcile.Request{
NamespacedName: types.NamespacedName{
Namespace: obj.GetNamespace(),
Name: ref.Name,
},
},
})
return
}
}
}

// byLabelEventHandler enqueues primary objects by evaluating label templates against
// the changed related object and listing primaries matching the resulting label selector.
type byLabelEventHandler struct {
clusterName string
client ctrlruntimeclient.Client
primaryDummy *unstructured.Unstructured
labelTemplates map[string]string
log *zap.SugaredLogger
}

func (h *byLabelEventHandler) Create(ctx context.Context, evt event.TypedCreateEvent[*unstructured.Unstructured], q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
h.enqueue(ctx, evt.Object, q)
}

func (h *byLabelEventHandler) Update(ctx context.Context, evt event.TypedUpdateEvent[*unstructured.Unstructured], q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
h.enqueue(ctx, evt.ObjectNew, q)
}

func (h *byLabelEventHandler) Delete(ctx context.Context, evt event.TypedDeleteEvent[*unstructured.Unstructured], q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
h.enqueue(ctx, evt.Object, q)
}

func (h *byLabelEventHandler) Generic(ctx context.Context, evt event.TypedGenericEvent[*unstructured.Unstructured], q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
h.enqueue(ctx, evt.Object, q)
}

func (h *byLabelEventHandler) enqueue(ctx context.Context, obj *unstructured.Unstructured, q workqueue.TypedRateLimitingInterface[mcreconcile.Request]) {
// Build the template context using the changed related object.
data := map[string]any{
"watchObject": map[string]any{
"name": obj.GetName(),
"namespace": obj.GetNamespace(),
"labels": obj.GetLabels(),
},
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please create an explicit struct for this templating context in https://github.com/kcp-dev/api-syncagent/blob/main/internal/sync/templating/related.go, which helps in documenting the available template variables. Please also add a corresponding New...() func, to ensure that the newly introduced struct always gets built correctly.

You can then extend https://github.com/kcp-dev/api-syncagent/blob/main/docs/content/publish-resources/templating.md, which is trivial once the struct is setup.


// Evaluate each label template to build the selector.
matchingLabels := ctrlruntimeclient.MatchingLabels{}
for key, tpl := range h.labelTemplates {
value, err := templating.Render(tpl, data)
if err != nil {
h.log.Warnw("Failed to evaluate byLabel template", "key", key, "template", tpl, "error", err)
return
}
matchingLabels[key] = value
}

// List primary objects matching the derived label selector.
primaryList := &unstructured.UnstructuredList{}
primaryList.SetAPIVersion(h.primaryDummy.GetAPIVersion())
primaryList.SetKind(h.primaryDummy.GetKind() + "List")

if err := h.client.List(ctx, primaryList, matchingLabels); err != nil {
h.log.Warnw("Failed to list primary objects for byLabel watch", "selector", matchingLabels, "error", err)
return
}

for i := range primaryList.Items {
primary := &primaryList.Items[i]
q.Add(mcreconcile.Request{
ClusterName: h.clusterName,
Request: reconcile.Request{
NamespacedName: types.NamespacedName{
Namespace: primary.GetNamespace(),
Name: primary.GetName(),
},
},
})
}
}
30 changes: 30 additions & 0 deletions sdk/apis/syncagent/v1alpha1/published_resource.go
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,36 @@ type RelatedResourceSpec struct {
// Mutation configures optional transformation rules for the related resource.
// Status mutations are only performed when the related resource originates in kcp.
Mutation *ResourceMutationSpec `json:"mutation,omitempty"`

// Watch configures how the agent identifies the owning primary object when a related
// resource with origin: kcp changes. When set, the agent sets up a watch on the related
// resource type and uses the configured rule to enqueue the correct primary object.
// Without this field, changes to origin:kcp related resources do not trigger reconciliation.
Watch *RelatedResourceWatch `json:"watch,omitempty"`
}

// RelatedResourceWatch configures how the watch handler maps a changed related resource
// back to its owning primary object.
// Exactly one of ByOwner or ByLabel must be set.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an XValidation rule to ensure this. There are some examples in this file already.

type RelatedResourceWatch struct {
// ByOwner configures the watch handler to inspect the OwnerReferences of the changed
// object. When an OwnerReference with the given Kind is found, the referenced owner
// is enqueued as the primary object.
// +optional
ByOwner *RelatedResourceWatchByOwner `json:"byOwner,omitempty"`

// ByLabel configures the watch handler to list primary objects matching a label selector
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rename this to "ByLabels" or "BySelector"? "ByLabel" is singular and I think this feature supports a selector with multiple key-value pairs, right?

// derived from the changed object. Each map key is a label key on the primary object;
// each value is a Go template expression evaluated with the changed object available as
// .watchObject (with fields .name, .namespace, .labels).
// +optional
ByLabel map[string]string `json:"byLabel,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we support a full-blown metav1.LabelSelector here?

}

// RelatedResourceWatchByOwner configures reverse lookup via OwnerReferences.
type RelatedResourceWatchByOwner struct {
// Kind is the Kind to look for in the OwnerReferences of the changed related object.
Kind string `json:"kind"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should take a full GVK here and not just compare based on Kinds.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think the opposite now. This is not required at all, is it?

We already know the GVK of the primary object on both sides (origin and destination). We also already know the GVK/GVR of every related resource (like ConfigMaps). So what the agent needs to do is to watch all the GVR of related resources on the origin side (ConfigMaps, for example) and check if they have an ownerRef to the GVK of the primary object (a Cluster object, or whatever). And if so, cool, enqueue the primary object (i.e. the owner).

We don't need any further configuration for the ByOwner functionality here, I think.

}

// RelatedResourceProjection describes how the source GVK of a related resource (i.e.
Expand Down
47 changes: 47 additions & 0 deletions sdk/apis/syncagent/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading