Fix events.k8s.io permission denial through APIExport virtual workspace by cnvergence · Pull Request #3894 · kcp-dev/kcp

cnvergence · 2026-03-10T12:10:37Z

Summary

When a controller created events through an APIExport virtual workspace, the event recorder
would get a permanent 403 "Server rejected event (will not retry!)" error.

Fixed the issue where APIExports claim events under the core group (""), but
client-go's event recorder uses events.k8s.io/v1. The virtual workspace APIReconciler
now registers both API groups when either event group is claimed.

The authorizer, maximal permission policy, and admission layers now normalise events.k8s.io/events to core ""/events when matching against permission claims, so both API groups are treated as equivalent.
When a new workspace is created, the APIBinding's Status.APIExportClusterName is not yet populated by the controller.
Both the authorizer and admission layers now fall back to matching via Spec.Reference.Export.Path when the status field is empty.

What Type of PR Is This?

/kind bug

Related Issue(s)

Fixes #3803

Release Notes

Fixed events.k8s.io permission denial through APIExport virtual workspace.

kcp-ci-bot · 2026-03-10T12:10:42Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

…ding Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

…d /apis/events.k8s.io/v1/events) get registered in the vws Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

cnvergence · 2026-03-11T09:48:10Z

/retest

mjudeikis · 2026-03-16T09:46:07Z

@cnvergence is it ready to merge?

cnvergence · 2026-03-16T09:57:20Z

Fixed all the issues, did some testing around as well, looks good to me
Hope it does for the reviewer as well

Logs from MCR controller:

2026-03-16T10:55:39+01:00	INFO	entrypoint	Setting up manager
2026-03-16T10:55:39+01:00	INFO	entrypoint	Starting manager
2026-03-16T10:55:39+01:00	INFO	controller-runtime.metrics	Starting metrics server
2026-03-16T10:55:39+01:00	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
2026-03-16T10:55:39+01:00	INFO	kcp-apiexport-cluster-provider	starting factory
2026-03-16T10:55:39+01:00	INFO	Starting Controller	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap"}
2026-03-16T10:55:39+01:00	INFO	Starting workers	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "worker count": 1}
2026-03-16T10:55:45+01:00	INFO	kcp-apiexport-cluster-provider	detected new object	{"object": {"metadata":{"name":"examples-apiexport-multicluster","uid":"7f1aef10-9493-4e60-be89-51f6e9764bfd","resourceVersion":"566","generation":1,"creationTimestamp":"2026-03-16T09:55:45Z","annotations":{"kcp.io/cluster":"root"},"ownerReferences":[{"apiVersion":"apis.kcp.io/v1alpha1","kind":"APIExport","name":"examples-apiexport-multicluster","uid":"6f5afeb9-0dba-4d13-a7de-7d328cdf650d"}],"managedFields":[{"manager":"kcp","operation":"Update","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"6f5afeb9-0dba-4d13-a7de-7d328cdf650d\"}":{}}},"f:spec":{".":{},"f:export":{".":{},"f:name":{},"f:path":{}}}}}]},"spec":{"export":{"path":"root","name":"examples-apiexport-multicluster"}},"status":{}}}
2026-03-16T10:55:45+01:00	INFO	kcp-apiexport-cluster-provider	detected updated object	{"object": {"metadata":{"name":"examples-apiexport-multicluster","uid":"7f1aef10-9493-4e60-be89-51f6e9764bfd","resourceVersion":"570","generation":1,"creationTimestamp":"2026-03-16T09:55:45Z","annotations":{"kcp.io/cluster":"root"},"ownerReferences":[{"apiVersion":"apis.kcp.io/v1alpha1","kind":"APIExport","name":"examples-apiexport-multicluster","uid":"6f5afeb9-0dba-4d13-a7de-7d328cdf650d"}],"managedFields":[{"manager":"root","operation":"Apply","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{}},"subresource":"status"},{"manager":"kcp","operation":"Update","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"6f5afeb9-0dba-4d13-a7de-7d328cdf650d\"}":{}}},"f:spec":{".":{},"f:export":{".":{},"f:name":{},"f:path":{}}}}}]},"spec":{"export":{"path":"root","name":"examples-apiexport-multicluster"}},"status":{}}}
2026-03-16T10:55:45+01:00	INFO	kcp-apiexport-cluster-provider	detected updated object	{"object": {"metadata":{"name":"examples-apiexport-multicluster","uid":"7f1aef10-9493-4e60-be89-51f6e9764bfd","resourceVersion":"575","generation":1,"creationTimestamp":"2026-03-16T09:55:45Z","annotations":{"kcp.io/cluster":"root"},"ownerReferences":[{"apiVersion":"apis.kcp.io/v1alpha1","kind":"APIExport","name":"examples-apiexport-multicluster","uid":"6f5afeb9-0dba-4d13-a7de-7d328cdf650d"}],"managedFields":[{"manager":"root","operation":"Apply","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{}},"subresource":"status"},{"manager":"kcp","operation":"Update","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"6f5afeb9-0dba-4d13-a7de-7d328cdf650d\"}":{}}},"f:spec":{".":{},"f:export":{".":{},"f:name":{},"f:path":{}}}}},{"manager":"kcp","operation":"Update","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{}}},"subresource":"status"}]},"spec":{"export":{"path":"root","name":"examples-apiexport-multicluster"}},"status":{"conditions":[{"type":"APIExportValid","status":"True","lastTransitionTime":"2026-03-16T09:55:45Z"},{"type":"PartitionValid","status":"True","lastTransitionTime":"2026-03-16T09:55:45Z"}]}}}
2026-03-16T10:55:46+01:00	INFO	kcp-apiexport-cluster-provider	detected updated object	{"object": {"metadata":{"name":"examples-apiexport-multicluster","uid":"7f1aef10-9493-4e60-be89-51f6e9764bfd","resourceVersion":"628","generation":1,"creationTimestamp":"2026-03-16T09:55:45Z","annotations":{"kcp.io/cluster":"root"},"ownerReferences":[{"apiVersion":"apis.kcp.io/v1alpha1","kind":"APIExport","name":"examples-apiexport-multicluster","uid":"6f5afeb9-0dba-4d13-a7de-7d328cdf650d"}],"managedFields":[{"manager":"root","operation":"Apply","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:endpoints":{"k:{\"url\":\"https://192.168.1.45:6443/services/apiexport/root/examples-apiexport-multicluster\"}":{".":{},"f:url":{}}}}},"subresource":"status"},{"manager":"kcp","operation":"Update","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"6f5afeb9-0dba-4d13-a7de-7d328cdf650d\"}":{}}},"f:spec":{".":{},"f:export":{".":{},"f:name":{},"f:path":{}}}}},{"manager":"kcp","operation":"Update","apiVersion":"apis.kcp.io/v1alpha1","time":"2026-03-16T09:55:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{}}},"subresource":"status"}]},"spec":{"export":{"path":"root","name":"examples-apiexport-multicluster"}},"status":{"conditions":[{"type":"APIExportValid","status":"True","lastTransitionTime":"2026-03-16T09:55:45Z"},{"type":"PartitionValid","status":"True","lastTransitionTime":"2026-03-16T09:55:45Z"}],"endpoints":[{"url":"https://192.168.1.45:6443/services/apiexport/root/examples-apiexport-multicluster"}]}}}
2026-03-16T10:55:46+01:00	INFO	Starting EventSource	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "func source: 0x1055dd540"}
2026-03-16T10:55:46+01:00	DEBUG	kind source handler registered	{"cluster": "1glx80kz6wqaccyk", "source": "kind", "hasRegistration": true}
2026-03-16T10:55:46+01:00	DEBUG	kind source cache synced	{"cluster": "1glx80kz6wqaccyk", "source": "kind"}
2026-03-16T10:55:46+01:00	INFO	Reconciling ConfigMap	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "reconcileID": "f5d5260c-1ffc-499a-bb14-5f7591187baf", "cluster": "1glx80kz6wqaccyk", "name": "kube-root-ca.crt", "uuid": "2376831a-f448-44f2-8c72-7b5147ade22e"}
2026-03-16T10:55:46+01:00	INFO	Reconciling ConfigMap	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "reconcileID": "f83fab42-6829-427a-b1ea-bc8ebff581ea", "cluster": "1glx80kz6wqaccyk", "name": "kube-root-ca.crt", "uuid": "fbafd202-1d6c-4f38-80c3-8b8a0dc6a965"}
2026-03-16T10:55:46+01:00	DEBUG	kcp-apiexport-cluster-provider.recorder-manager	ConfigMap kube-root-ca.crt reconciled	{"url": "https://192.168.1.45:6443/services/apiexport/root/examples-apiexport-multicluster", "type": "Normal", "object": "nil", "action": "ConfigMap Reconciled", "reason": "normal"}
2026-03-16T10:55:46+01:00	DEBUG	kcp-apiexport-cluster-provider.recorder-manager	ConfigMap kube-root-ca.crt reconciled	{"url": "https://192.168.1.45:6443/services/apiexport/root/examples-apiexport-multicluster", "type": "Normal", "object": "nil", "action": "ConfigMap Reconciled", "reason": "normal"}
2026-03-16T10:55:46+01:00	INFO	Starting EventSource	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "func source: 0x1055dd540"}
2026-03-16T10:55:46+01:00	DEBUG	kind source handler registered	{"cluster": "erc8bn3nmv0x5sji", "source": "kind", "hasRegistration": true}
2026-03-16T10:55:46+01:00	DEBUG	kind source cache synced	{"cluster": "erc8bn3nmv0x5sji", "source": "kind"}
2026-03-16T10:55:46+01:00	INFO	Reconciling ConfigMap	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "reconcileID": "6ef99b08-c0d5-4079-a301-2809c610f3f9", "cluster": "erc8bn3nmv0x5sji", "name": "kube-root-ca.crt", "uuid": "d9fe37cf-126c-4b2f-b6b6-06cb6f09aa74"}
2026-03-16T10:55:46+01:00	DEBUG	kcp-apiexport-cluster-provider.recorder-manager	ConfigMap kube-root-ca.crt reconciled	{"url": "https://192.168.1.45:6443/services/apiexport/root/examples-apiexport-multicluster", "type": "Normal", "object": "nil", "action": "ConfigMap Reconciled", "reason": "normal"}
2026-03-16T10:55:46+01:00	INFO	Reconciling ConfigMap	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "reconcileID": "4755f517-b534-40d0-b898-29fe5cec1749", "cluster": "erc8bn3nmv0x5sji", "name": "kube-root-ca.crt", "uuid": "442dd653-da64-4418-889e-5f8f68388c85"}
2026-03-16T10:55:46+01:00	DEBUG	kcp-apiexport-cluster-provider.recorder-manager	ConfigMap kube-root-ca.crt reconciled	{"url": "https://192.168.1.45:6443/services/apiexport/root/examples-apiexport-multicluster", "type": "Normal", "object": "nil", "action": "ConfigMap Reconciled", "reason": "normal"}
2026-03-16T10:55:46+01:00	INFO	Starting EventSource	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "func source: 0x1055dd540"}
2026-03-16T10:55:46+01:00	DEBUG	kind source handler registered	{"cluster": "1zymsjl7u8j6o5wl", "source": "kind", "hasRegistration": true}
2026-03-16T10:55:46+01:00	DEBUG	kind source cache synced	{"cluster": "1zymsjl7u8j6o5wl", "source": "kind"}
2026-03-16T10:55:46+01:00	INFO	Reconciling ConfigMap	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "reconcileID": "5bdb9dd3-11f7-4496-acfc-b7cf08c52c8c", "cluster": "1zymsjl7u8j6o5wl", "name": "kube-root-ca.crt", "uuid": "e34ef8fe-5b3d-4b22-bbe8-895bbfdc25b1"}
2026-03-16T10:55:46+01:00	DEBUG	kcp-apiexport-cluster-provider.recorder-manager	ConfigMap kube-root-ca.crt reconciled	{"url": "https://192.168.1.45:6443/services/apiexport/root/examples-apiexport-multicluster", "type": "Normal", "object": "nil", "action": "ConfigMap Reconciled", "reason": "normal"}
2026-03-16T10:55:46+01:00	INFO	Reconciling ConfigMap	{"controller": "kcp-configmap-controller", "controllerGroup": "", "controllerKind": "ConfigMap", "reconcileID": "58ad962a-5fa1-42a3-8b7b-eea2bcd01679", "cluster": "1zymsjl7u8j6o5wl", "name": "kube-root-ca.crt", "uuid": "d0228d22-6452-40aa-832b-12a67ad0ae02"}
2026-03-16T10:55:46+01:00	DEBUG	kcp-apiexport-cluster-provider.recorder-manager	ConfigMap kube-root-ca.crt reconciled	{"url": "https://192.168.1.45:6443/services/apiexport/root/examples-apiexport-multicluster", "type": "Normal", "object": "nil", "action": "ConfigMap Reconciled", "reason": "normal"}

mjudeikis · 2026-03-16T09:58:28Z

/lgtm
/approve

lets see

kcp-ci-bot · 2026-03-16T09:58:33Z

LGTM label has been added.

Details

Git tree hash: d26aa46ae15fd603ecc6084ec5ef9f97f7e01112

kcp-ci-bot · 2026-03-16T09:58:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mjudeikis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mjudeikis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cnvergence · 2026-03-19T13:58:10Z

/retest

cnvergence · 2026-03-19T15:11:05Z

/retest

kcp-ci-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 10, 2026

cnvergence marked this pull request as ready for review March 10, 2026 12:17

kcp-ci-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 10, 2026

cnvergence added 5 commits March 10, 2026 14:07

Check if permission claims are accepted when reconciling binding

1cf30d3

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

Add guard and failback checks in admission and authorizer for API Bin…

f177d46

…ding Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

Make normalizerGR global

e4345b1

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

when either event group is claimed, both API paths (/api/v1/events an…

97c4e1d

…d /apis/events.k8s.io/v1/events) get registered in the vws Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

fix lint

3db5f39

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com> On-behalf-of: @SAP karol.szwaj@sap.com

cnvergence force-pushed the fix-apiexport-event branch from 8794915 to 3db5f39 Compare March 10, 2026 13:11

kcp-ci-bot assigned mjudeikis Mar 16, 2026

kcp-ci-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 16, 2026

kcp-ci-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 16, 2026

kcp-ci-bot merged commit f9ed0d3 into kcp-dev:main Mar 19, 2026
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix events.k8s.io permission denial through APIExport virtual workspace#3894

Fix events.k8s.io permission denial through APIExport virtual workspace#3894
kcp-ci-bot merged 5 commits intokcp-dev:mainfrom
cnvergence:fix-apiexport-event

cnvergence commented Mar 10, 2026

Uh oh!

kcp-ci-bot commented Mar 10, 2026

Uh oh!

cnvergence commented Mar 11, 2026

Uh oh!

mjudeikis commented Mar 16, 2026

Uh oh!

cnvergence commented Mar 16, 2026

Uh oh!

mjudeikis commented Mar 16, 2026

Uh oh!

kcp-ci-bot commented Mar 16, 2026

Uh oh!

kcp-ci-bot commented Mar 16, 2026

Uh oh!

cnvergence commented Mar 19, 2026

Uh oh!

cnvergence commented Mar 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cnvergence commented Mar 10, 2026

Summary

What Type of PR Is This?

Related Issue(s)

Release Notes

Uh oh!

kcp-ci-bot commented Mar 10, 2026

Uh oh!

cnvergence commented Mar 11, 2026

Uh oh!

mjudeikis commented Mar 16, 2026

Uh oh!

cnvergence commented Mar 16, 2026

Uh oh!

mjudeikis commented Mar 16, 2026

Uh oh!

kcp-ci-bot commented Mar 16, 2026

Uh oh!

kcp-ci-bot commented Mar 16, 2026

Uh oh!

cnvergence commented Mar 19, 2026

Uh oh!

cnvergence commented Mar 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants