diff --git a/config/_default/hugo.yaml b/config/_default/hugo.yaml index a115ee5..8727f74 100644 --- a/config/_default/hugo.yaml +++ b/config/_default/hugo.yaml @@ -164,7 +164,12 @@ menu: pre: weight: 16 + - name: Resources + url: /project/resources/ + pre: + weight: 17 + - name: Project url: /project/ pre: - weight: 17 + weight: 18 diff --git a/content/en/docs/operating/admission-policies.md b/content/en/docs/operating/admission-policies.md index d913e3a..bcc4b7d 100644 --- a/content/en/docs/operating/admission-policies.md +++ b/content/en/docs/operating/admission-policies.md @@ -747,6 +747,114 @@ You may consider the upstream policies, depending on your needs: * [QoS Guaranteed](https://kyverno.io/policies/other/require-qos-guaranteed/require-qos-guaranteed/) +## Certificates + +### Deny ClusterIssuer in Certificates + +Often when working in multi-tenant environments, you want to ensure that tenants are not using `ClusterIssuers` to issue certificates, but rather use namespaced `Issuers` within their own namespace. This policy enforces that `cert-manager.io/v1/Certificate` resources do not reference `ClusterIssuers` and that the `Issuer` referenced is in the same namespace as the `Certificate`. + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="Kyverno" >}} +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: certificates-only-local-issuer +spec: + validationFailureAction: Enforce + background: true + rules: + - name: deny-clusterissuer-in-certificates + match: + any: + - resources: + kinds: + - cert-manager.io/v1/Certificate + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + validate: + message: "Certificates must not reference ClusterIssuers; use a namespaced Issuer in the same namespace." + deny: + conditions: + any: + - key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}" + operator: Equals + value: "ClusterIssuer" + + - name: deny-cross-namespace-issuerref-in-certificates + match: + any: + - resources: + kinds: + - cert-manager.io/v1/Certificate + validate: + message: "Certificates must reference an Issuer in the same namespace (spec.issuerRef.namespace must be empty or equal to the Certificate namespace)." + deny: + conditions: + any: + # If issuerRef.namespace is set and differs from the Certificate namespace -> deny + - key: "{{request.object.spec.issuerRef.namespace || '' }}" + operator: NotEquals + value: "" + # AND also not equal to request namespace + - key: "{{ request.object.spec.issuerRef.namespace || request.namespace }}" + operator: NotEquals + value: "{{ request.namespace }}"{{< /tab >}} +{{% /tabpane %}} + +### Deny ClusterIssuer in Gateways + +Deny to usage of ClusterIssuers in Gateways by checking for the `cert-manager.io/cluster-issuer` annotation. This ensures that tenants use namespaced issuer mechanisms instead. + +This requires extra permissions to allow Kyverno to read Gateway resources: + +```yaml +admissionController: + rbac: + clusterRole: + extraResources: + - apiGroups: ["gateway.networking.k8s.io"] + resources: ["*"] + verbs: ["get", "list", "watch"] +``` + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="Kyverno" >}} +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: gateways-deny-cluster-issuer-annotation +spec: + validationFailureAction: Enforce + background: false + rules: + - name: deny-cert-manager-cluster-issuer-annotation + match: + any: + - resources: + kinds: + - gateway.networking.k8s.io/v1/Gateway + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + validate: + message: "Gateways must not use cert-manager.io/cluster-issuer; use namespaced issuer mechanisms instead." + deny: + conditions: + any: + - key: "{{ request.object.metadata.annotations.\"cert-manager.io/cluster-issuer\" || '' }}" + operator: NotEquals + value: ""{{< /tab >}} +{{% /tabpane %}} + + + ## Images ### Allowed Registries diff --git a/content/en/docs/operating/architecture.md b/content/en/docs/operating/architecture.md index 9611a32..bd33dfc 100644 --- a/content/en/docs/operating/architecture.md +++ b/content/en/docs/operating/architecture.md @@ -30,32 +30,56 @@ In Capsule, we introduce a new persona called the `Tenant Owner`. The goal is to ### Capsule Administrators +[Configure Capsule Administrators](/docs/operating/setup/configuration/#administrators). The ClusterRoles assigned to Administrators can be configured in the [CapsuleConfiguration](/docs/operating/setup/configuration/#rbac) as well. + They are promoted to [Tenant-Owners](#tenant-owners) for all available tenants. Effectively granting them the ability to manage all namespaces within the cluster, across all tenants. **Note**: Granting Capsule Administrator rights should be done with caution, as it provides extensive control over the cluster's multi-tenant environment. **When granting Capsule Administrator rights, the entity gets the privileges to create any namespace (also not part of capsule tenants) and the privileges to delete any tenant namespaces.** Capsule Administrators can: + - Create and manage [namespaces via labels in any tenant](/docs/tenants/namespaces/#label). - Create namespaces outside of tenants. - Delete namespaces in any tenant. Administrators come in handy in bootstrap scenarios or GitOps scenarios where certain users/serviceaccounts need to be able to manage namespaces for all tenants. -[Configure Capsule Administrators](/docs/operating/setup/configuration/#administrators) - ### Capsule Users +[Configure Capsule Users](/docs/operating/setup/configuration/#users) + Any entity which needs to interact with tenants and their namespaces must be defined as a **Capsule User**. This is where the flexibility of Capsule comes into play. You can define users or groups as Capsule Users, allowing them to create and manage namespaces within any tenant they have access to. If they are not defined as Capsule Users, any interactions will be ignored by Capsule. Often a best practice is to define a single group which identifies all your tenant users. This way you can have one generic group for all your users and then use additional groups to separate responsibilities (e.g. administrators vs normal users). **Only one entry is needed to identify a Capsule User. This is only important for Namespace Admission.**. ![Capsule Users Admission](/images/content/capsule-users-admission.drawio.png) -[Configure Capsule Users](/docs/operating/setup/configuration/#users) +You can always verify the effective Capsule Users by checking the Configuration Status. As this is variable with [Tenant Owners](#tenant-owners), the status will always show the effective users: + +```bash +kubectl get capsuleconfiguration default -o jsonpath='{.status.users}' | jq + +[ + { + "kind": "Group", + "name": "oidc:kubernetes:admin" + }, + { + "kind": "Group", + "name": "projectcapsule.dev" + }, + { + "kind": "User", + "name": "test-user" + } +] +``` ### Tenant Owners -**Every Tenant Owner must be a [Capsule User](#capsule-users)** +[Configure Tenant Owners](/docs/tenants/permissions/#ownership) + +**Every Tenant Owner must be a [Capsule User](#capsule-users). By using the [TenantOwner CRD](/docs/tenants/permissions/#tenant-owners) this is automatically handeled.** They manage the namespaces within their tenants and perform administrative tasks confined to their tenant boundaries. This delegation allows teams to operate more autonomously while still adhering to organizational policies. Tenant Owners can be used to shift reposnsability of one tenant towards this user group. promoting them to the SPOC of all namespaces within the tenant. @@ -67,8 +91,6 @@ Tenant Owners can: Capsule provides robust tools to strictly enforce tenant boundaries, ensuring that each tenant operates within its defined limits. This separation of duties promotes both security and efficient resource management. -[Configure Tenant Owners](/docs/tenants/permissions/#ownership) - ## Layouts Let's dicuss different Tenant Layouts which could be used . These are just approaches we have seen, however you might also find a combination of these which fits your use-case. diff --git a/content/en/docs/operating/best-practices/_index.md b/content/en/docs/operating/best-practices/_index.md index 537442d..32bacdd 100644 --- a/content/en/docs/operating/best-practices/_index.md +++ b/content/en/docs/operating/best-practices/_index.md @@ -1,7 +1,7 @@ --- title: Best Practices weight: 2 -description: Best Practices when running Capsule in production +description: Best Practices when running Capsule in production --- @@ -30,7 +30,7 @@ Instead, a centralized secrets management system should be established — such To integrate these external secret stores with Kubernetes, the [External Secrets Operator (ESO)](https://external-secrets.io/latest/) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing. If no external secret store is available, there should at least be a secure way to store sensitive data in Git. -In our ecosystem, we provide a solution based on SOPS (Secrets OPerationS) for this use case. +In our ecosystem, we provide a solution based on SOPS (Secrets OPerationS) for this use case; called the [sops-operator](https://github.com/peak-scale/sops-operator). [👉 Demonstration](https://killercoda.com/peakscale/course/playgrounds/sops-secrets) diff --git a/content/en/docs/operating/monitoring.md b/content/en/docs/operating/monitoring.md index a017abe..667633d 100644 --- a/content/en/docs/operating/monitoring.md +++ b/content/en/docs/operating/monitoring.md @@ -33,6 +33,215 @@ Dashboard which grants a detailed overview over the ResourcePools Example rules to give you some idea, what's possible. 1. Alert on [ResourcePools](../resourcepools/) usage + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: capsule-resourcepools-alerts +spec: +groups: + - name: capsule-resourcepools.rules + rules: + - alert: CapsuleResourcePoolHighUsageWarning + expr: | + capsule_pool_usage_percentage > 90 + for: 10m + labels: + severity: warning + annotations: + summary: High resource usage in Resourcepool + description: | + Resource {{ $labels.resource }} in pool {{ $labels.pool }} is at {{ $value }}% usage for the last 10 minutes. + + - alert: CapsuleResourcePoolHighUsageCritical + expr: | + capsule_pool_usage_percentage > 95 + for: 10m + labels: + severity: critical + annotations: + summary: Critical resource usage in Resourcepool + description: | + Resource {{ $labels.resource }} in pool {{ $labels.pool }} has exceeded 95% usage for the last 10 minutes. + + - alert: CapsuleResourcePoolExhausted + expr: | + capsule_pool_condition{condition="Exhausted"} == 1 + for: 60m + labels: + severity: critical + annotations: + summary: Resource pool exhausted + description: | + Pool {{ $labels.pool }} has been Exhausted for more than 60 minutes. + + - alert: CapsuleResourcePoolNotReady + expr: | + capsule_pool_condition{condition="Ready"} == 0 + for: 10m + labels: + severity: warning + annotations: + summary: Resource pool not ready + description: | + Pool {{ $labels.pool }} has not been Ready for more than 10 minutes. + + + - name: capsule-resourcepoolclaims.rules + rules: + - alert: CapsuleResourcePoolClaimExhausted + expr: | + capsule_claim_condition{condition="Exhausted"} == 1 + for: 24h + labels: + severity: critical + annotations: + summary: ResourcePoolClaim exhausted + description: | + ResourcePoolClaim {{ $labels.name }}/{{ $labels.target_namespace }} has been Exhausted for more than 24 hours. + + - alert: CapsuleResourcePoolClaimNotReady + expr: | + capsule_claim_condition{condition="Ready"} == 0 + for: 60m + labels: + severity: warning + annotations: + summary: ResourcePoolClaim not ready + description: | + ResourcePoolClaim {{ $labels.name }}/{{ $labels.target_namespace }} has not been Ready for more than 60 minutes. +``` + +--- + +### Metrics + +The following Metrics are exposed and can be used for monitoring: + +```shell + +# HELP capsule_claim_condition The current condition status of a claim. +# TYPE capsule_claim_condition gauge +capsule_claim_condition{condition="Bound",name="get-me-customer",target_namespace="solar-test"} 1 +capsule_claim_condition{condition="Bound",name="get-me-solar",target_namespace="solar-test"} 1 +capsule_claim_condition{condition="Bound",name="get-me-solar-2",target_namespace="solar-test"} 0 +capsule_claim_condition{condition="Exhausted",name="get-me-customer",target_namespace="solar-test"} 0 +capsule_claim_condition{condition="Exhausted",name="get-me-solar",target_namespace="solar-test"} 0 +capsule_claim_condition{condition="Exhausted",name="get-me-solar-2",target_namespace="solar-test"} 1 +capsule_claim_condition{condition="Ready",name="get-me-customer",target_namespace="solar-test"} 1 +capsule_claim_condition{condition="Ready",name="get-me-solar",target_namespace="solar-test"} 1 +capsule_claim_condition{condition="Ready",name="get-me-solar-2",target_namespace="solar-test"} 1 + +# HELP capsule_claim_pool The current assigned pool of a claim. +# TYPE capsule_claim_pool gauge +capsule_claim_pool{name="get-me-solar",pool="solar-compute",target_namespace="solar-test"} 1 +capsule_claim_pool{name="get-me-solar-2",pool="solar-compute",target_namespace="solar-test"} 1 + +# HELP capsule_claim_resource The given amount of resources from the claim +# TYPE capsule_claim_resource gauge +capsule_claim_resource{name="compute",resource="limits.cpu",target_namespace="solar-prod"} 0.375 +capsule_claim_resource{name="compute",resource="limits.memory",target_namespace="solar-prod"} 4.02653184e+08 +capsule_claim_resource{name="compute",resource="requests.cpu",target_namespace="solar-prod"} 0.375 +capsule_claim_resource{name="compute",resource="requests.memory",target_namespace="solar-prod"} 4.02653184e+08 +capsule_claim_resource{name="compute-10",resource="limits.memory",target_namespace="solar-prod"} 1.073741824e+10 +capsule_claim_resource{name="compute-2",resource="limits.cpu",target_namespace="solar-prod"} 0.5 +capsule_claim_resource{name="compute-2",resource="limits.memory",target_namespace="solar-prod"} 5.36870912e+08 +capsule_claim_resource{name="compute-2",resource="requests.cpu",target_namespace="solar-prod"} 0.5 +capsule_claim_resource{name="compute-2",resource="requests.memory",target_namespace="solar-prod"} 5.36870912e+08 +capsule_claim_resource{name="compute-3",resource="requests.cpu",target_namespace="solar-prod"} 0.5 +capsule_claim_resource{name="compute-4",resource="requests.cpu",target_namespace="solar-test"} 0.5 +capsule_claim_resource{name="compute-5",resource="requests.cpu",target_namespace="solar-test"} 0.5 +capsule_claim_resource{name="compute-6",resource="requests.cpu",target_namespace="solar-test"} 5 +capsule_claim_resource{name="pods",resource="pods",target_namespace="solar-test"} 3 + +# HELP capsule_pool_available Current resource availability for a given resource in a resource pool +# TYPE capsule_pool_available gauge +capsule_pool_available{pool="solar-compute",resource="limits.cpu"} 1.125 +capsule_pool_available{pool="solar-compute",resource="limits.memory"} 1.207959552e+09 +capsule_pool_available{pool="solar-compute",resource="requests.cpu"} 0.125 +capsule_pool_available{pool="solar-compute",resource="requests.memory"} 1.207959552e+09 +capsule_pool_available{pool="solar-size",resource="pods"} 4 + + +# HELP capsule_pool_condition Current conditions for a given resource in a resource pool +# TYPE capsule_pool_condition gauge +capsule_pool_condition{condition="Exhausted",pool="solar-size"} 0 +capsule_pool_condition{condition="Exhausted",pool="solar-compute"} 1 +capsule_pool_condition{condition="Ready",pool="solar-size"} 1 +capsule_pool_condition{condition="Ready",pool="solar-compute"} 1 + +# HELP capsule_pool_exhaustion Resources become exhausted, when there's not enough available for all claims and the claims get queued +# TYPE capsule_pool_exhaustion gauge +capsule_pool_exhaustion{pool="solar-compute",resource="limits.memory"} 1.073741824e+10 +capsule_pool_exhaustion{pool="solar-compute",resource="requests.cpu"} 5.5 + +# HELP capsule_pool_exhaustion_percentage Resources become exhausted, when there's not enough available for all claims and the claims get queued (Percentage) +# TYPE capsule_pool_exhaustion_percentage gauge +capsule_pool_exhaustion_percentage{pool="solar-compute",resource="limits.memory"} 788.8888888888889 +capsule_pool_exhaustion_percentage{pool="solar-compute",resource="requests.cpu"} 4300 + +# HELP capsule_pool_limit Current resource limit for a given resource in a resource pool +# TYPE capsule_pool_limit gauge +capsule_pool_limit{pool="solar-compute",resource="limits.cpu"} 2 +capsule_pool_limit{pool="solar-compute",resource="limits.memory"} 2.147483648e+09 +capsule_pool_limit{pool="solar-compute",resource="requests.cpu"} 2 +capsule_pool_limit{pool="solar-compute",resource="requests.memory"} 2.147483648e+09 +capsule_pool_limit{pool="solar-size",resource="pods"} 7 + +# HELP capsule_pool_namespace_usage Current resources claimed on namespace basis for a given resource in a resource pool for a specific namespace +# TYPE capsule_pool_namespace_usage gauge +capsule_pool_namespace_usage{pool="solar-compute",resource="limits.cpu",target_namespace="solar-prod"} 0.875 +capsule_pool_namespace_usage{pool="solar-compute",resource="limits.memory",target_namespace="solar-prod"} 9.39524096e+08 +capsule_pool_namespace_usage{pool="solar-compute",resource="requests.cpu",target_namespace="solar-prod"} 1.375 +capsule_pool_namespace_usage{pool="solar-compute",resource="requests.cpu",target_namespace="solar-test"} 0.5 +capsule_pool_namespace_usage{pool="solar-compute",resource="requests.memory",target_namespace="solar-prod"} 9.39524096e+08 +capsule_pool_namespace_usage{pool="solar-size",resource="pods",target_namespace="solar-test"} 3 + +# HELP capsule_pool_namespace_usage_percentage Current resources claimed on namespace basis for a given resource in a resource pool for a specific namespace (percentage) +# TYPE capsule_pool_namespace_usage_percentage gauge +capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="limits.cpu",target_namespace="solar-prod"} 43.75 +capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="limits.memory",target_namespace="solar-prod"} 43.75 +capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="requests.cpu",target_namespace="solar-prod"} 68.75 +capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="requests.cpu",target_namespace="solar-test"} 25 +capsule_pool_namespace_usage_percentage{pool="solar-compute",resource="requests.memory",target_namespace="solar-prod"} 43.75 +capsule_pool_namespace_usage_percentage{pool="solar-size",resource="pods",target_namespace="solar-test"} 42.857142857142854 + +# HELP capsule_pool_resource Type of resource being used in a resource pool +# TYPE capsule_pool_resource gauge +capsule_pool_resource{pool="solar-compute",resource="limits.cpu"} 1 +capsule_pool_resource{pool="solar-compute",resource="limits.memory"} 1 +capsule_pool_resource{pool="solar-compute",resource="requests.cpu"} 1 +capsule_pool_resource{pool="solar-compute",resource="requests.memory"} 1 +capsule_pool_resource{pool="solar-size",resource="pods"} 1 + +# HELP capsule_pool_usage Current resource usage for a given resource in a resource pool +# TYPE capsule_pool_usage gauge +capsule_pool_usage{pool="solar-compute",resource="limits.cpu"} 0.875 +capsule_pool_usage{pool="solar-compute",resource="limits.memory"} 9.39524096e+08 +capsule_pool_usage{pool="solar-compute",resource="requests.cpu"} 1.875 +capsule_pool_usage{pool="solar-compute",resource="requests.memory"} 9.39524096e+08 +capsule_pool_usage{pool="solar-size",resource="pods"} 3 + +# HELP capsule_pool_usage_percentage Current resource usage for a given resource in a resource pool (percentage) +# TYPE capsule_pool_usage_percentage gauge +capsule_pool_usage_percentage{pool="solar-compute",resource="limits.cpu"} 43.75 +capsule_pool_usage_percentage{pool="solar-compute",resource="limits.memory"} 43.75 +capsule_pool_usage_percentage{pool="solar-compute",resource="requests.cpu"} 93.75 +capsule_pool_usage_percentage{pool="solar-compute",resource="requests.memory"} 43.75 +capsule_pool_usage_percentage{pool="solar-size",resource="pods"} 42.857142857142854 +``` + +## Replications + +Instrumentation for [Replications](../replications/). + +### Rules + +Example rules to give you some idea, what's possible. + +1. Alert on [Replications](../replications/) usage + ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule @@ -169,6 +378,8 @@ capsule_pool_usage_percentage{pool="solar-compute",resource="requests.memory"} 4 capsule_pool_usage_percentage{pool="solar-size",resource="pods"} 42.857142857142854 ``` + + ## Tenants Instrumentation for [Tenants](../tenants/). diff --git a/content/en/docs/operating/setup/configuration.md b/content/en/docs/operating/setup/configuration.md index b2b15a8..cddb6e3 100644 --- a/content/en/docs/operating/setup/configuration.md +++ b/content/en/docs/operating/setup/configuration.md @@ -1,6 +1,6 @@ --- -title: Controller Options -weight: 100 +title: Configuration +weight: 2 description: > Understand the Capsule configuration options and how to use them. --- @@ -118,35 +118,97 @@ manager: allowServiceAccountPromotion: true ``` -### `userGroups` +### `cacheInvalidation` -{{% alert title="Information" color="info" %}} -**Deprecated**: This option is deprecated and will be removed in future releases. [Please use the `users` option to specify both user and group owners for Capsule tenancy](#users). -{{% /alert %}} +The reconcile periode caches are invalidated. Invalidation is already attempted when resources change, however in certain scenarios it might be necessary to do out of order cache invalidations to ensure proper garbage collection of resources. -Names of the groups for Capsule users. Users must have this group to be considered for the Capsule tenancy. If a user does not have any group mentioned here, they are not recognized as a Capsule user. +```yaml +manager: + options: + cacheInvalidation: 0h30m0s +``` + +### `rbac` + +Define configurations for the RBAC which is being managed and applied by Capsule. ```yaml manager: options: - capsuleUserGroups: - - system:serviceaccounts:tenants-gitops - - company:org:users + rbac: + # -- The ClusterRoles applied for Administrators + adminitrationClusterRoles: + - capsule-namespace-deleter + + # -- The ClusterRoles applied for ServiceAccounts which had owner Promotion + promotionClusterRoles: + - capsule-namespace-provisioner + - capsule-namespace-deleter + + # -- Name for the ClusterRole required to grant Namespace Deletion permissions. + deleter: capsule-namespace-deleter + + # -- Name for the ClusterRole required to grant Namespace Provision permissions. + provisioner: capsule-namespace-provisioner ``` -### `userNames` +### `impersonation` + +For Replications by default the controller ServiceAccount is used to perform the operations. However it is possible to define a dedicated ServiceAccount to be used for that purpose. Within this configuration you can define properties such as the endpoint of the kube-apiserver and if service account promotion should be allowed for this client. Also declare default service account to be used for replication operations. By default the `https://kubernetes.default.svc` endpoint is used. + +```yaml +manager: + options: + impersonation: + # Kubernetes API Endpoint to use for the operations + endpoint: "https://capsule-proxy.capsule-system.svc:8081" + + # Toggles if TLS verification for the endpoint is performed or not + skipTlsVerify: false + + # Key in the secret that holds the CA certificate (e.g., "ca.crt") + caSecretKey: "ca.crt" + + # Name of the secret containing the CA certificate + caSecretName: "capsule-proxy-tls" + + # Namespace where the CA certificate secret is located + caSecretNamespace: "capsule-system" + + # Default ServiceAccount for global resources (GlobalTenantResource) [Cluster Scope] + # When defined, users are required to use this ServiceAccount anywhere in the cluster + # unless they explicitly provide their own. Once this is set, Capsule will add this ServiceAccount + # for all GlobalTenantResources, if they don't already have a ServiceAccount defined. + globalDefaultServiceAccount: "capsule-global-sa" + + # Namespace of the for the ServiceAccount provided by the globalDefaultServiceAccount property + globalDefaultServiceAccountNamespace: "tenant-system" + + # Default ServiceAccount for tenant resources (TenantResource) [Namespaced Scope] + # When defined, users are required to use this ServiceAccount anywhere in the cluster + # unless they explicitly provide their own. Once this is set, Capsule will add this ServiceAccount + # for all GlobalTenantResources, if they don't already have a ServiceAccount defined. + tenantDefaultServiceAccount: "default" +``` -{{% alert title="Information" color="info" %}} -**Deprecated**: This option is deprecated and will be removed in future releases. [Please use the `users` option to specify both user and group owners for Capsule tenancy](#users). -{{% /alert %}} +### `admission` -Names of the users for Capsule users. Users must have this name to be considered for the Capsule tenancy. If userGroups are set, the properties are ORed, meaning that a user can be recognized as a Capsule user if they have one of the groups or one of the names. +Configuration for the dynamic admission webhooks used by Capsule for mutating and validating requests. The settings are used from the static webhook configurations created during installation of Capsule and abstracted by the helm chart ```yaml manager: options: - userNames: - - system:serviceaccount:crossplane-system:crossplane-k8s-provider + admission: + mutating: + client: + caBundle: cert + url: https://172.24.52.212:9443 + name: capsule-dynamic + validating: + client: + caBundle: cert + url: https://172.24.52.212:9443 + name: capsule-dynamic ``` ## Controller Options diff --git a/content/en/docs/operating/setup/installation.md b/content/en/docs/operating/setup/installation.md index 9fccba7..c363ac9 100644 --- a/content/en/docs/operating/setup/installation.md +++ b/content/en/docs/operating/setup/installation.md @@ -6,24 +6,24 @@ description: "Installing the Capsule Controller" ## Requirements - * [Helm 3](https://github.com/helm/helm/releases) is required when installing the Capsule Operator chart. Follow Helm’s official for installing helm on your particular operating system. - * A Kubernetes cluster 1.16+ with following [Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) enabled: + * [Helm 3](https://github.com/helm/helm/releases) is required when installing the Capsule Operator chart. Follow Helm’s official documentation for installing Helm on your operating system. + * A Kubernetes cluster (v1.16+) with the following [Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) enabled: * PodNodeSelector * LimitRanger * ResourceQuota * MutatingAdmissionWebhook * ValidatingAdmissionWebhook * A [Kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file accessing the Kubernetes cluster with cluster admin permissions. - * [Cert-Manager](https://cert-manager.io/) is recommended but not required + * [Cert-Manager](https://cert-manager.io/) is required by default but can be disabled. It is used to manage the TLS certificates for the Capsule Admission Webhooks. ## Installation -We officially only support the installation of Capsule using the Helm chart. The chart itself handles the Installation/Upgrade of needed [CustomResourceDefinitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). The following Artifacthub repository are official: +We officially only support the installation of Capsule using the Helm chart. The chart itself handles the installation/upgrade of the required [CustomResourceDefinitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). The following Artifact Hub repositories are official: -* [Artifacthub Page (OCI)](https://artifacthub.io/packages/helm/capsule/capsule) -* [Artifacthub Page (Legacy - Best Effort)](https://artifacthub.io/packages/helm/projectcapsule/capsule) +* [Artifact Hub Page (OCI)](https://artifacthub.io/packages/helm/capsule/capsule) +* [Artifact Hub Page (Legacy - Best Effort)](https://artifacthub.io/packages/helm/projectcapsule/capsule) -Perform the following steps to install the capsule Operator: +Perform the following steps to install the Capsule operator: 1. Add repository: @@ -54,31 +54,94 @@ Perform the following steps to install the capsule Operator: helm uninstall capsule -n capsule-system -## Considerations +## Production Here are some key considerations to keep in mind when installing Capsule. Also check out the **[Best Practices](/docs/operating/best-practices)** for more information. +### Strict RBAC + + +{{% alert title="Attention" color="warning" %}} +Ensure to first upgrade to version `0.13.0` of capsule before enabling strict mode. As it requires fields which are newly added with version `0.13.0`. +{{% /alert %}} + + +By default, the Capsule controller runs with the ClusterRole `cluster-admin`, which provides full access to the cluster. This is because the controller itself must grant RoleBindings on a per-namespace basis that by default reference the ClusterRole `admin`, which needs to at least match the permissions of the controller itself. However, for production environments we recommend configuring stricter RBAC permissions for the Capsule controller. You can enable the minimal required permissions by setting the following value in the Helm chart: + +```yaml +manager: + rbac: + strict: true +``` + +This grants the controller the minimal permissions required for its own operation. However, that alone is not sufficient for it to function properly. The ClusterRole for the controller allows aggregating further permissions to it via the following labels: + +* `projectcapsule.dev/aggregate-to-controller: "true"` +* `projectcapsule.dev/aggregate-to-controller-instance: {{ .Release.Name }}` + +In other words, you must aggregate all ClusterRoles that are assigned to [Tenant owners](/docs/tenants/permissions/#owner-roles) or used for [additional RoleBindings](/docs/tenants/permissions/#strict). This applies only to ClusterRoles that are not managed by Capsule (see [Configuration](/docs/operating/setup/configuration/#rbac)). By default, the only such ClusterRole granted to owners is `admin` (not managed by Capsule). + +```bash +kubectl label clusterrole admin projectcapsule.dev/aggregate-to-controller=true +``` + +Verify that the label has been applied: + +```yaml +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: admin + labels: + projectcapsule.dev/aggregate-to-controller: "true" +rules: +... +``` + +If you are missing permissions you will see an error status for the respective tenants reflecting + +```bash +kubectl get tnt +NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STATUS AGE +green Active 2 False cannot sync rolebindings items: rolebindings.rbac.authorization.k8s.io "capsule:managed:658936e7f2a30e35" is forbidden: user "system:serviceaccount:capsule-system:capsule" (groups=["system:serviceaccounts" "system:serviceaccounts:capsule-system" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:... 5s + +``` + +Alternatively, you can enable only the minimal required permissions by setting the following value in the Helm chart: + +```yaml +manager: + rbac: + minimal: true +``` + +Before you enable this option, you must implement the required permissions for your use case. Depending on which features you are using, you may need to take manual action, for example: + +* [Migrate additional RoleBindings](/docs/tenants/permissions/#strict) + + + ### Admission Policies While Capsule provides a robust framework for managing multi-tenancy in Kubernetes, it does not include built-in admission policies for enforcing specific security or operational standards for all possible aspects of a Kubernetes cluster. [We provide additional policy recommendations here](/docs/operating/admission-policies/). ### Certificate Management -We recommend using [cert-manager](https://cert-manager.io/) to manage the TLS certificates for Capsule. This will ensure that your Capsule installation is secure and that the certificates are automatically renewed. Capsule requires a valid TLS certificate for it's Admission Webserver. By default Capsule reconciles it's own TLS certificate. To use cert-manager, you can set the following values: +By default, Capsule delegates its certificate management to [cert-manager](https://cert-manager.io/). This is the recommended way to manage the TLS certificates for Capsule. However, you can also use Capsule's built-in TLS reconciler to manage the certificates. This is not recommended for production environments. To enable the TLS reconciler, use the following values: ```yaml certManager: - generateCertificates: true + generateCertificates: false tls: - enableController: false - create: false + enableController: true + create: true ``` ### Webhooks -Capsule makes use of [webhooks for admission control](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers). Ensure that your cluster supports webhooks and that they are properly configured. The webhooks are automatically created by Capsule during installation. However some of these webhooks will cause problems when capsule is not running (this is especially problematic in single-node clusters). Here are the webhooks you need to watch out for. +Capsule makes use of [webhooks for admission control](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers). Ensure that your cluster supports webhooks and that they are properly configured. The webhooks are automatically created by Capsule during installation. However, some of these webhooks will cause problems when Capsule is not running (this is especially problematic in single-node clusters). Here are the webhooks you need to watch out for. -Generally we recommend to use [matchconditions](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchconditions) for all the webhooks to avoid problems when Capsule is not running. You should exclude your system critical components from the Capsule webhooks. For namespaced resources (`pods`, `services`, etc.) the webhooks all select only namespaces which are part of a Capsule Tenant. If your system critical components are not part of a Capsule Tenant, they will not be affected by the webhooks. However, if you have system critical components which are part of a Capsule Tenant, you should exclude them from the Capsule webhooks by using [matchconditions](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchconditions) as well or add more specific [namespaceselectors](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector)/[objectselectors](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-objectselector) to exclude them. This can also be considered to improve performance. +Generally, we recommend using [matchConditions](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchconditions) for all webhooks to avoid problems when Capsule is not running. You should exclude your system-critical components from the Capsule webhooks. For namespaced resources (`pods`, `services`, etc.) the webhooks select only namespaces that are part of a Capsule Tenant. If your system-critical components are not part of a Capsule Tenant, they will not be affected by the webhooks. However, if you have system-critical components that are part of a Capsule Tenant, you should exclude them from the Capsule webhooks by using [matchConditions](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchconditions) as well, or add more specific [namespaceSelectors](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector)/[objectSelectors](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-objectselector) to exclude them. This can also improve performance. [Refer to the webhook values](https://artifacthub.io/packages/helm/projectcapsule/capsule#webhooks-parameters). @@ -86,7 +149,7 @@ Generally we recommend to use [matchconditions](https://kubernetes.io/docs/refer #### Nodes -There is a webhook which catches interactions with the Node resource. This Webhook is mainly interesting, when you make use of [Node Metadata](/docs/tenants/enforcement/#nodes). In any other case it will just case you problems. By default the webhook is **disabled**, but you can enabled it by setting the following value: +There is a webhook which catches interactions with the Node resource. This webhook is mainly relevant when you make use of [Node metadata](/docs/tenants/enforcement/#nodes). In most other cases, it will only cause problems. By default, the webhook is **disabled**, but you can enable it by setting the following value: ```yaml webhooks: @@ -104,7 +167,7 @@ webhooks: failurePolicy: Ignore ``` -If you still want to use the feature, you could execlude the kube-system namespace (or any other namespace you want to exclude) from the webhook by setting the following value: +If you still want to use the feature, you could exclude the kube-system namespace (or any other namespace you want to exclude) from the webhook by setting the following value: ```yaml webhooks: @@ -119,7 +182,7 @@ webhooks: #### Namespaces -Namespaces are the most important resource in Capsule. The Namespace Webhook is responsible for enforcing the Capsule Tenant boundaries. It is enabled by default and should not be disabled. However, you may change the matchConditions to execlude certain namespaces from the Capsule Tenant boundaries. For example, you can exclude the kube-system namespace by setting the following value: +Namespaces are the most important resource in Capsule. The Namespace webhook is responsible for enforcing the Capsule Tenant boundaries. It is enabled by default and should not be disabled. However, you may change the matchConditions to exclude certain namespaces from the Capsule Tenant boundaries. For example, you can exclude the kube-system namespace by setting the following value: ```yaml webhooks: @@ -130,12 +193,36 @@ webhooks: expression: '!("system:serviceaccounts:kube-system" in request.userInfo.groups)' ``` +#### Protected + +By default resources with the following values are protected by a webhook to be changed by [Capsule Users]: + +```yaml +webhooks: + hooks: + managed: + objectSelector: + matchExpressions: + - key: "projectcapsule.dev/created-by" + operator: In + values: + - "controller" + - "resources" + - key: "projectcapsule.dev/managed-by" + operator: In + values: + - "controller" +``` + + ## GitOps There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource. ### ArgoCD +Visit the [ArgoCD Integration](/ecosystem/integrations/argocd/) for more options to integrate Capsule with ArgoCD. + Manifests to get you started with ArgoCD. For ArgoCD you might need to skip the validation of the `CapsuleConfiguration` resources, otherwise there might be errors on the first install: {{% alert title="Information" color="warning" %}} @@ -169,11 +256,6 @@ spec: valuesObject: crds: install: true - certManager: - generateCertificates: true - tls: - enableController: false - create: false manager: options: annotations: @@ -181,9 +263,11 @@ spec: capsuleConfiguration: default ignoreUserGroups: - oidc:administators - capsuleUserGroups: - - oidc:kubernetes-users - - system:serviceaccounts:capsule-argo-addon + users: + - kind: Group + name: oidc:kubernetes-users + - kind: Group + name: system:serviceaccounts:tenants-system monitoring: dashboards: enabled: true @@ -191,22 +275,6 @@ spec: enabled: true annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true - proxy: - enabled: true - webhooks: - enabled: true - certManager: - generateCertificates: true - options: - generateCertificates: false - oidcUsernameClaim: "email" - extraArgs: - - "--feature-gates=ProxyClusterScoped=true" - - "--feature-gates=ProxyAllNamespaced=true" - serviceMonitor: - enabled: true - annotations: - argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true destination: server: https://kubernetes.default.svc @@ -275,36 +343,21 @@ spec: values: crds: install: true - certManager: - generateCertificates: true - tls: - enableController: false - create: false manager: options: capsuleConfiguration: default ignoreUserGroups: - oidc:administators - capsuleUserGroups: - - oidc:kubernetes-users - - system:serviceaccounts:capsule-argo-addon + users: + - kind: Group + name: oidc:kubernetes-users + - kind: Group + name: system:serviceaccounts:tenants-system monitoring: dashboards: enabled: true serviceMonitor: enabled: true - proxy: - enabled: true - webhooks: - enabled: true - certManager: - generateCertificates: true - options: - generateCertificates: false - oidcUsernameClaim: "email" - extraArgs: - - "--feature-gates=ProxyClusterScoped=true" - - "--feature-gates=ProxyAllNamespaced=true" --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository @@ -331,13 +384,13 @@ To verify artifacts you need to have [cosign installed](https://github.com/sigst # Helm Chart export COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule -To verify the signature of the docker image, run the following command. Replace `` with an [available release tag](https://github.com/projectcapsule/capsule/pkgs/container/capsule): +To verify the signature of the docker image, run the following command. COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign verify ghcr.io/projectcapsule/capsule: \ --certificate-identity-regexp="https://github.com/projectcapsule/capsule/.github/workflows/docker-publish.yml@refs/tags/*" \ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" | jq -To verify the signature of the helm image, run the following command. Replace `` with an [available release tag](https://github.com/projectcapsule/capsule/pkgs/container/charts%2Fcapsule): +To verify the signature of the helm image, run the following command. COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign verify ghcr.io/projectcapsule/charts/capsule: \ --certificate-identity-regexp="https://github.com/projectcapsule/capsule/.github/workflows/helm-publish.yml@refs/tags/*" \ @@ -347,22 +400,20 @@ To verify the signature of the helm image, run the following command. Replace `< Capsule creates and attests to the provenance of its builds using the [SLSA standard](https://slsa.dev/spec/v0.2/provenance) and meets the [SLSA Level 3](https://slsa.dev/spec/v0.1/levels) specification. The attested provenance may be verified using the cosign tool. -Verify the provenance of the docker image. Replace `` with an [available release tag](https://github.com/projectcapsule/capsule/pkgs/container/capsule) +Verify the provenance of the docker image. -```bash +``` cosign verify-attestation --type slsaprovenance \ --certificate-identity-regexp="https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/*" \ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \ - ghcr.io/projectcapsule/capsule: | jq .payload -r | base64 --decode | jq + ghcr.io/projectcapsule/capsule:{{< capsule_chart_version >}} | jq .payload -r | base64 --decode | jq ``` -Verify the provenance of the helm image. Replace `` with an [available release tag](https://github.com/projectcapsule/capsule/pkgs/container/charts%2Fcapsule) - ```bash cosign verify-attestation --type slsaprovenance \ --certificate-identity-regexp="https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/*" \ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \ - ghcr.io/projectcapsule/charts/capsule: | jq .payload -r | base64 --decode | jq + ghcr.io/projectcapsule/charts/capsule:{{< capsule_chart_version >}} | jq .payload -r | base64 --decode | jq ``` ### Software Bill of Materials (SBOM) @@ -376,13 +427,13 @@ An SBOM (Software Bill of Materials) in CycloneDX JSON format is published for e export COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule -To inspect the SBOM of the docker image, run the following command. Replace `` with an [available release tag](https://github.com/projectcapsule/capsule/pkgs/container/capsule): +To inspect the SBOM of the docker image, run the following command. - COSIGN_REPOSITORY=ghcr.io/projectcapsule/capsule cosign download sbom ghcr.io/projectcapsule/capsule: + COSIGN_REPOSITORY=ghcr.io/projectcapsule/capsule cosign download sbom ghcr.io/projectcapsule/capsule:{{< capsule_chart_version >}} -To inspect the SBOM of the helm image, run the following command. Replace `` with an [available release tag](https://github.com/projectcapsule/capsule/pkgs/container/charts%2Fcapsule): +To inspect the SBOM of the helm image, run the following command. - COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign download sbom ghcr.io/projectcapsule/charts/capsule: + COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign download sbom ghcr.io/projectcapsule/charts/capsule:{{< capsule_chart_version >}} ## Compatibility diff --git a/content/en/docs/operating/templating.md b/content/en/docs/operating/templating.md new file mode 100644 index 0000000..76bea5a --- /dev/null +++ b/content/en/docs/operating/templating.md @@ -0,0 +1,81 @@ +--- +title: Templating +weight: 10 +description: "Templating in Capsule Items" +--- + + +## Fast Templates + +For simple template cases we provide a fast templating engine. With this engine, you can use Go templates syntax to reference Tenant and Namespace fields. There are no operators or anything else supported. + +Available fields are: + + * `{{tenant.name}}`: The Name of the Tenant + * `{{namespace}}`: The Name of the Tenant + + +## Sprout Templating + +Our template library is mainly based on the upstream implementation from Sprout. You can find the all available functions here: + +* [https://docs.atom.codes/sprout/registries/list-of-all-registries](https://docs.atom.codes/sprout/registries/list-of-all-registries) + +We have removed certain functions which could exploit runtime information. Therefor the following functions are not available: + + * `env` + * `expandEnv` + +### Data + +You can provide structured data for each `Tenant` which can be used in templating: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + data: + bool: true + foo: bar + list: + - a + - b + number: 123 + obj: + nested: value +``` + +### Custom Functions + +Custom Functions we provide in our template package. + +#### deterministicUUID + +`deterministicUUID` generates a deterministic, RFC-4122–compliant UUID (version 5 + RFC4122 variant) from a set of input strings. It is designed for use in templates where you need stable, repeatable IDs derived from meaningful inputs (e.g. cluster name, tenant, role name), instead of random UUIDs. + +This is especially useful for: + + * Crossplane / IaC resources that must not change IDs across reconciles + +The function takes any number of strings and turns them into a UUID in a fully deterministic way. + +What that means in practice: + +* If you call it twice with the same values, you get the same UUID +* If any input changes, the UUID changes too +* There is no randomness involved +* The output is always a valid UUID + +So from the outside, it behaves just like a normal UUID — just deterministic. + +```go +deterministicUUID(parts ...string) string +``` + +Example usage: + +```yaml +{{ deterministicUUID "cluster-a" "app-123" "tenant-x" "some-role" }} +``` \ No newline at end of file diff --git a/content/en/docs/overview/_index.md b/content/en/docs/overview/_index.md index 3202320..1fe99b5 100644 --- a/content/en/docs/overview/_index.md +++ b/content/en/docs/overview/_index.md @@ -1,37 +1,55 @@ --- title: Overview weight: 2 -description: Understand the problem Capsule is attempting to solve and how it works +description: Run multiple teams on a single Kubernetes cluster - with strong isolation, self-service, and zero cluster sprawl. --- -Capsule implements a multi-tenant, policy-based environment in your Kubernetes cluster. It is designed as a microservices-based ecosystem with a minimalist approach, leveraging only upstream Kubernetes. +Capsule is a Kubernetes Operator that turns a single cluster into a shared, multi-tenant platform. Teams get their own isolated space - called a **Tenant** - with their own namespaces, resource budgets, and policies. Cluster administrators maintain full control, while teams work autonomously without stepping on each other. -With Capsule, you have an ecosystem that addresses the challenges of hosting multiple parties on a shared Kubernetes cluster. Let's look at a typical scenario for using Capsule. +No custom Kubernetes distribution. No extra tooling your users need to learn. Just plain Kubernetes, made shareable. -
+## The Problem -![capsule-workflow](/images/content/capsule-architecture.drawio.png) +Kubernetes namespaces provide a basic level of isolation, but they have no hierarchy. As soon as multiple teams or customers need to share a cluster, you face hard choices: -As shown, we can create a new boundary between Kubernetes (cluster) administrators and tenant audiences. While Kubernetes administrators define the boundaries of a tenant, the tenant audience can act within the namespaces of that tenant. For the tenant audience, we differentiate between Tenant Owners and Tenant Users. The main advantage Tenant Owners are granted is the ability to create namespaces within the tenants they own. This achieves a shift-left approach: instead of depending on Kubernetes administrators to create namespaces, Tenant Owners can manage this themselves, thereby granting them greater autonomy within strictly defined boundaries. +- **Isolation is all-or-nothing** - there is no native way to group namespaces per team or enforce consistent policies across them. +- **Namespace sprawl** - cluster admins become a bottleneck, manually creating and configuring every namespace. +- **Cluster sprawl** - organizations spin up a separate cluster per team to achieve proper isolation, multiplying operational overhead across the board. +## How Capsule Works -## What's the problem with the current status? +Capsule introduces the **Tenant** - a lightweight, cluster-scoped resource that groups one or more Kubernetes namespaces under a shared set of boundaries. -Kubernetes introduces the Namespace object type to create logical partitions of the cluster as isolated slices. However, when implementing advanced multi-tenancy scenarios, this soon becomes complicated because of the flat structure of Kubernetes namespaces and the impossibility of sharing resources among namespaces belonging to the same tenant. To overcome this, cluster admins tend to provision a dedicated cluster for each group of users, teams, or departments. As an organization grows, the number of clusters to manage and keep aligned becomes an operational nightmare, described as the well-known phenomenon of cluster sprawl. +Everything defined on a Tenant is automatically inherited by all its namespaces: -## Entering Capsule +- **RBAC bindings** - roles are propagated to every namespace without manual setup. +- **Resource quotas & limits** - CPU, memory, and storage budgets managed at the tenant level via [Resource Pools](/docs/resourcepools/). +- **Admission rules** - allowed image registries, pull policies, security contexts, and more. +- **Templated resource distribution** - using [Replications](/docs/replications/), resources such as NetworkPolicies, ImagePullSecrets, and LimitRanges are automatically distributed into every namespace a Tenant Owner creates, using Go templates for dynamic values like namespace name or tenant name. -Capsule takes a different approach. In a single cluster, the Capsule Controller aggregates multiple namespaces in a lightweight abstraction called a Tenant, which is basically a grouping of Kubernetes namespaces. Within each tenant, users are free to create their namespaces and share all the assigned resources. +![capsule-workflow](/images/content/capsule-architecture.drawio.png) -On the other side, the Capsule Policy Engine keeps the different tenants isolated from each other. Network and security policies, resource quotas, limit ranges, RBAC, and other policies defined at the tenant level are automatically inherited by all the namespaces in the tenant. Users are then free to operate their tenants autonomously, without intervention from the cluster administrator. +![capsule-operator](/images/content/capsule-operator.svg) -
+## Who Does What -![capsule-operator](/images/content/capsule-operator.svg) +| Role | Responsibility | +|---|---| +| **Cluster Admin** | Installs Capsule, creates Tenants, sets resource budgets and policies. Never a bottleneck for day-to-day namespace work. | +| **Tenant Owner** | Creates and manages namespaces within their Tenant. Assigns access to team members. No cluster-level permissions needed. | +| **Tenant User** | Deploys workloads inside tenant namespaces, within the limits the owner has set. | + +This shift-left model means Tenant Owners handle day-to-day namespace operations themselves, freeing cluster admins from repetitive provisioning work. +## Key Features -## What problems are out of scope +- **[Tenants & Namespaces](/docs/tenants/)** - Group namespaces into logical units per team, product, or customer. Policy inheritance is automatic. +- **[Resource Pools](/docs/resourcepools/)** - Distribute CPU, memory, and storage budgets across namespaces with flexible claiming rather than fixed per-namespace quotas. +- **[Replications](/docs/replications/)** - Propagate Kubernetes resources (Secrets, ConfigMaps, etc.) across tenant namespaces automatically. +- **[Policy Rules](/docs/tenants/rules/)** - Enforce allowed registries, pull policies, and namespace metadata requirements on a per-tenant basis. +- **[Capsule Proxy](/docs/proxy/)** - Let users run `kubectl get namespaces` and see only their own, without granting cluster-wide LIST permissions. Also works for other cluster wide requests, like `kubectl get pods -A`, or listing Persistent Volumes that are used by a Persistent Volume Claim inside the tenant. -Capsule does not aim to solve the following problems: +## Get Started -* Handling of Custom Resource Definition management. Capsule does not aim to manage the control of Custom Resource Definition. Users have to implement their own solution. +- [**Quickstart** - create your first Tenant in minutes](/docs/tenants/quickstart/) +- [**Tenant Docs** - explore everything Tenants can do](/docs/tenants/) diff --git a/content/en/docs/proxy/gangplank.md b/content/en/docs/proxy/gangplank.md new file mode 100644 index 0000000..18118ab --- /dev/null +++ b/content/en/docs/proxy/gangplank.md @@ -0,0 +1,114 @@ +--- +title: Gangplank +description: Capsule Integration with Gangplank +weight: 5 +--- + +[Gangplank](https://github.com/sighupio/gangplank) is a web application that allows users to authenticate with an OIDC provider and configure their kubectl configuration file with the [OpenID Connect Tokens](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens). Gangplank is based on [Gangway](https://github.com/vmware-archive/gangway), which is no longer maintained. + +## Prerequisites + +For Authentication you will need a Confidential OIDC client configured in your OIDC provider, such as [Keycloak](https://www.keycloak.org/), [Dex](https://dexidp.io/), or [Google Cloud Identity](https://docs.github.com/en/actions/how-tos/secure-your-work/security-harden-deployments/oidc-in-google-cloud-platform). By default the Kubernetes API only validates tokens against a Public OIDC client, so you will need to configure your OIDC provider to allow the Gangplank client to issue tokens. You must make use of the Kubernetes Authentication Configuration, which allows to define multiple audiences (clients). This way we can issue tokens for a gangplank client, which is Confidential, and a kubernetes client, which is Public. The Kubernetes API will validate the tokens against both clients. The Config might look like this: + +```yaml +apiVersion: apiserver.config.k8s.io/v1beta1 +kind: AuthenticationConfiguration +jwt: +- issuer: + url: https://keycloak/realms/realm-name + audiences: + - kubernetes + - gangplank + audienceMatchPolicy: MatchAny # This one is important + claimMappings: + username: + claim: 'email' + prefix: "" + groups: + claim: 'groups' + prefix: "" +``` + +[Read More](/docs/operating/authentication/#configuring-kubernetes-api-server) + +## Integration + +We provide the option to install Gangplank alongside the Capsule Proxy. This allows users to authenticate with their OIDC provider and configure their kubectl configuration file with the OpenID Connect Tokens, which are valid for the Capsule Proxy Ingress. This way users can access the Kubernetes API through the Capsule Proxy without having to worry about the authentication and token management. To install gangplank, you must enable it: + +```yaml +gangplank: + enabled: true +``` + +Gangplank won't just work out of the box. You will need to provide some configuration values, which are required for gangplank to work properly. These values are: + +* `GANGPLANK_CONFIG_AUTHORIZE_URL`: `https://keycloak/realms/realm-name/protocol/openid-connect/auth` +* `GANGPLANK_CONFIG_TOKEN_URL`: `https://keycloak/realms/realm-name/protocol/openid-connect/token` +* `GANGPLANK_CONFIG_REDIRECT_URL`: `https://gangplank.example.com/callback` +* `GANGPLANK_CONFIG_CLIENT_ID`: `gangplank` +* `GANGPLANK_CONFIG_CLIENT_SECRET`: `` +* `GANGPLANK_CONFIG_USERNAME_CLAIM`: The JWT claim to use as the username. (we use `email` in the authentication config above, so this should also be `email`) +* `GANGPLANK_CONFIG_APISERVER_URL`: The URL **Capsule Proxy Ingress**. Since the users probably want to access the Kubernetes API from outside the cluster, you should use the Capsule Proxy Ingress URL here. + +When using the Helm chart, you can set these values in the `values.yaml` file: + +```yaml +gangplank: + enabled: true + config: + clusterName: "tenant-cluster" + apiServerURL: "https://capsule-proxy.company.com:443" + scopes: ["openid", "profile", "email", "groups", "offline_access"] + redirectURL: "https://gangplank.company.com/callback" + usernameClaim: "email" + clientID: "gangplank" + authorizeURL: "https://keycloak/realms/realm-name/protocol/openid-connect/auth" + tokenURL: "https://keycloak/realms/realm-name/protocol/openid-connect/token" + + # Mount The Client Secret as Environment Variables (GANGPLANK_CONFIG_CLIENT_SECRET) + envFrom: + - secretRef: + name: gangplank-secrets +``` + +Now the only thing left to do is to change the CA certificate which is provided. By default the CA certificate is set to the Kubernetes API server CA certificate, which is not valid for the Capsule Proxy Ingress. For this we can simply override the CA certificate in the Helm chart. You can do this by creating a Kubernetes Secret with the CA certificate and mounting it as a volume in the Gangplank deployment. + +```yaml +gangplank: + volumeMounts: + - mountPath: /var/run/secrets/kubernetes.io/serviceaccount + name: token-ca + volumes: + - name: token-ca + projected: + sources: + - serviceAccountToken: + path: token + - secret: + name: proxy-ingress-tls + items: + - key: tls.crt + path: ca.crt +``` + +**Note**: In this example we used the `tls.crt` key of the `proxy-ingress-tls` secret. This is a classic [Cert-Manager](https://cert-manager.io/) TLS secret, which contains only the Certificate and Key for the Capsule Proxy Ingress. However the Certificate contains the CA certificate as well (Certificate Chain), so we can use it to verify the Capsule Proxy Ingress. If you use a different secret, make sure to adjust the key accordingly. + +If that's not possible you can also set the CA certificate as an environment variable: + +```yaml +gangplank: + config: + clusterCAPath: "/capsule-proxy/ca.crt" + volumeMounts: + - mountPath: /capsule-proxy/ + name: token-ca + volumes: + - name: token-ca + projected: + sources: + - secret: + name: proxy-ingress-tls + items: + - key: tls.crt + path: ca.crt +``` \ No newline at end of file diff --git a/content/en/docs/proxy/installation.md b/content/en/docs/proxy/installation.md deleted file mode 100644 index c8bf7f3..0000000 --- a/content/en/docs/proxy/installation.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: Installation -description: > - Installation guide for the capsule-proxy -date: 2017-01-05 -weight: 1 ---- -Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources. - -The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard. - -We only support the installation via helm-chart, you can find the chart here: - -* [https://artifacthub.io/packages/helm/capsule-proxy/capsule-proxy](https://artifacthub.io/packages/helm/capsule-proxy/capsule-proxy) - -## Considerations - -Consdierations when deploying capsule-proxy - -### Exposure - -Depending on your environment, you can expose the capsule-proxy by: - - * `Ingress` - * `NodePort Service` - * `LoadBalance Service` - * `HostPort` - * `HostNetwork` - -Here how it looks like when exposed through an Ingress Controller: - -``` - +-----------+ +-----------+ +-----------+ - kubectl ------>|:443 |--------->|:9001 |-------->|:6443 | - +-----------+ +-----------+ +-----------+ - ingress-controller capsule-proxy kube-apiserver -``` - -### User Authentication - -The capsule-proxy intercepts all the requests from the kubectl client directed to the APIs Server. Users using a TLS client-based authentication with a certificate and key can talk with the API Server since it can forward client certificates to the Kubernetes APIs server. - -It is possible to protect the capsule-proxy using a certificate provided by Let's Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream. - -If your prerequisite is exposing capsule-proxy using an Ingress, you must rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server. - -### Distribute CA within the Cluster - -The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named `capsule-proxy` in the `capsule-system` namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required. - -Here's an example of how to distribute the CA certificate to the namespace `tekton-pipelines` by using `kubectl` and `jq`: - -```shell - kubectl get secret capsule-proxy -n capsule-system -o json \ - | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \ - | kubectl apply -n tekton-pipelines -f - -``` - -This can be used for development purposes, but it's not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments: - - * [Kubernetes Reflector](https://github.com/EmberStack/kubernetes-reflector) - - -### HTTP Support - -> NOTE: kubectl will not work against a http server. - -Capsule proxy supports `https` and `http`, although the latter is not recommended, we understand that it can be useful for some use cases (i.e. development, working behind a TLS-terminated reverse proxy and so on). As the default behaviour is to work with https, we need to use the flag --enable-ssl=false if we want to work under http. - -After having the capsule-proxy working under http, requests must provide authentication using an allowed Bearer Token. - -For example: - -```shell -TOKEN= -curl -H "Authorization: Bearer $TOKEN" http://localhost:9001/api/v1/namespaces -``` - -### Metrics - -Starting from the v0.3.0 release, Capsule Proxy exposes Prometheus metrics available at `http://0.0.0.0:8080/metrics`. - -The offered metrics are related to the internal controller-manager code base, such as work queue and REST client requests, and the Go runtime ones. - -Along with these, metrics capsule_proxy_response_time_seconds and capsule_proxy_requests_total have been introduced and are specific to the Capsule Proxy code-base and functionalities. - -capsule_proxy_response_time_seconds offers a bucket representation of the HTTP request duration. The available variables for these metrics are the following ones: - -path: the HTTP path of every single request that Capsule Proxy passes to the upstream -capsule_proxy_requests_total counts the global requests that Capsule Proxy is passing to the upstream with the following labels. - -path: the HTTP path of every single request that Capsule Proxy passes to the upstream -status: the HTTP status code of the request - - diff --git a/content/en/docs/proxy/options.md b/content/en/docs/proxy/options.md deleted file mode 100644 index 46d7ca5..0000000 --- a/content/en/docs/proxy/options.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: Controller Options -description: > - Configure the Capsule Proxy Controller -date: 2024-02-20 -weight: 3 ---- - -You can customize the Capsule Proxy with the following configuration - -## Flags - - - - -## Feature Gates - -Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available: - -| **Feature Gate** | **Default Value** | **Description** | -| :--- | :--- | :--- | -| `ProxyAllNamespaced` | `true` | `ProxyAllNamespaced` allows to proxy all the Namespaced objects. When enabled, it will discover apis and ensure labels are set for resources in all tenant namespaces resulting in increased memory. However this feature helps with user experience. | -| `SkipImpersonationReview` | `false` | `SkipImpersonationReview` allows to skip the impersonation review for all requests containing impersonation headers (user and groups). **DANGER:** Enabling this flag allows any user to impersonate as any user or group essentially bypassing any authorization. Only use this option in trusted environments where authorization/authentication is offloaded to external systems. | -| `ProxyClusterScoped` | `false` | `ProxyClusterScoped` allows to proxy all clusterScoped objects for all tenant users. These can be defined via [ProxySettings](./proxysettings) | diff --git a/content/en/docs/proxy/setup/_index.md b/content/en/docs/proxy/setup/_index.md new file mode 100644 index 0000000..27bded3 --- /dev/null +++ b/content/en/docs/proxy/setup/_index.md @@ -0,0 +1,6 @@ +--- +title: Setup +weight: 1 +description: > + Install and configure the Capsule Proxy +--- diff --git a/content/en/docs/proxy/setup/installation.md b/content/en/docs/proxy/setup/installation.md new file mode 100644 index 0000000..bd22543 --- /dev/null +++ b/content/en/docs/proxy/setup/installation.md @@ -0,0 +1,268 @@ +--- +title: Installation +description: > + Installation guide for the capsule-proxy +date: 2017-01-05 +weight: 1 +--- +Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources. + +The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard. + +We officially only support the installation of Capsule using the Helm chart. The chart itself handles the Installation/Upgrade of needed [CustomResourceDefinitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). The following Artifacthub repository are official: + +* [Artifacthub Page (OCI)](https://artifacthub.io/packages/helm/capsule-proxy/capsule-proxy) +* [Artifacthub Page (Legacy - Best Effort)](https://artifacthub.io/packages/helm/projectcapsule/capsule-proxy) + +Perform the following steps to install the capsule Operator: + +1. Add repository: + + helm repo add projectcapsule https://projectcapsule.github.io/charts + +2. Install Capsule-Proxy: + + helm install capsule-proxy projectcapsule/capsule-proxy -n capsule-system --create-namespace + + or (**OCI**) + + helm install capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy -n capsule-system --create-namespace + +3. Show the status: + + helm status capsule-proxy -n capsule-system + +4. Upgrade the Chart + + helm upgrade capsule-proxy projectcapsule/capsule-proxy -n capsule-system + + or (**OCI**) + + helm upgrade capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy --version 0.13.0 + +5. Uninstall the Chart + + helm uninstall capsule-proxy -n capsule-system + + +## GitOps + +There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource. + +### ArgoCD + +Visit the [ArgoCD Integration](/ecosystem/integrations/argocd/) for more options to integrate Capsule with ArgoCD. + +```yaml +--- +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: capsule + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: system + source: + repoURL: ghcr.io/projectcapsule/charts + targetRevision: {{< capsule_chart_version >}} + chart: capsule + helm: + valuesObject: + ... + proxy: + enabled: true + webhooks: + enabled: true + certManager: + generateCertificates: true + options: + generateCertificates: false + oidcUsernameClaim: "email" + extraArgs: + - "--feature-gates=ProxyClusterScoped=true" + serviceMonitor: + enabled: true + annotations: + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true + + destination: + server: https://kubernetes.default.svc + namespace: capsule-system + + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - ServerSideApply=true + - CreateNamespace=true + - PrunePropagationPolicy=foreground + - PruneLast=true + - RespectIgnoreDifferences=true + retry: + limit: 5 + backoff: + duration: 5s + factor: 2 + maxDuration: 3m +--- +apiVersion: v1 +kind: Secret +metadata: + name: capsule-repo + namespace: argocd + labels: + argocd.argoproj.io/secret-type: repository +stringData: + url: ghcr.io/projectcapsule/charts + name: capsule + project: system + type: helm + enableOCI: "true" +``` + +### FluxCD + +```yaml +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: capsule + namespace: flux-system +spec: + serviceAccountName: kustomize-controller + targetNamespace: "capsule-system" + interval: 10m + releaseName: "capsule" + chart: + spec: + chart: capsule + version: "{{< capsule_chart_version >}}" + sourceRef: + kind: HelmRepository + name: capsule + interval: 24h + install: + createNamespace: true + upgrade: + remediation: + remediateLastFailure: true + driftDetection: + mode: enabled + values: + proxy: + enabled: true + webhooks: + enabled: true + certManager: + generateCertificates: true + options: + generateCertificates: false + oidcUsernameClaim: "email" + extraArgs: + - "--feature-gates=ProxyClusterScoped=true" +--- +apiVersion: source.toolkit.fluxcd.io/v1 +kind: HelmRepository +metadata: + name: capsule + namespace: flux-system +spec: + type: "oci" + interval: 12h0m0s + url: oci://ghcr.io/projectcapsule/charts +``` + +## Considerations + +Consdierations when deploying capsule-proxy + +### Exposure + +Depending on your environment, you can expose the capsule-proxy by: + + * `Ingress` + * `NodePort Service` + * `LoadBalance Service` + * `HostPort` + * `HostNetwork` + +Here how it looks like when exposed through an Ingress Controller: + +``` + +-----------+ +-----------+ +-----------+ + kubectl ------>|:443 |--------->|:9001 |-------->|:6443 | + +-----------+ +-----------+ +-----------+ + ingress-controller capsule-proxy kube-apiserver +``` + +### User Authentication + +The capsule-proxy intercepts all the requests from the kubectl client directed to the APIs Server. Users using a TLS client-based authentication with a certificate and key can talk with the API Server since it can forward client certificates to the Kubernetes APIs server. + +It is possible to protect the capsule-proxy using a certificate provided by Let's Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream. + +If your prerequisite is exposing capsule-proxy using an Ingress, you must rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server. + +### Certificate Management + +By default, Capsule delegates its certificate management to cert-manager. This is the recommended way to manage the TLS certificates for Capsule.This relates to certifiacates for the proxy and the admissions server. However, you can also use a job to generate self-signed certificates and store them in a Kubernetes Secret: + +```yaml +options: + generateCertificates: true +certManager: + generateCertificates: false +``` + +#### Distribute CA within the Cluster + +The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named `capsule-proxy` in the `capsule-system` namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required. + +Here's an example of how to distribute the CA certificate to the namespace `tekton-pipelines` by using `kubectl` and `jq`: + +```shell + kubectl get secret capsule-proxy -n capsule-system -o json \ + | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \ + | kubectl apply -n tekton-pipelines -f - +``` + +This can be used for development purposes, but it's not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments: + + * [Kubernetes Reflector](https://github.com/EmberStack/kubernetes-reflector) + + +### HTTP Support + +> NOTE: kubectl will not work against a http server. + +Capsule proxy supports `https` and `http`, although the latter is not recommended, we understand that it can be useful for some use cases (i.e. development, working behind a TLS-terminated reverse proxy and so on). As the default behaviour is to work with https, we need to use the flag --enable-ssl=false if we want to work under http. + +After having the capsule-proxy working under http, requests must provide authentication using an allowed Bearer Token. + +For example: + +```shell +TOKEN= +curl -H "Authorization: Bearer $TOKEN" http://localhost:9001/api/v1/namespaces +``` + +### Metrics + +Starting from the v0.3.0 release, Capsule Proxy exposes Prometheus metrics available at `http://0.0.0.0:8080/metrics`. + +The offered metrics are related to the internal controller-manager code base, such as work queue and REST client requests, and the Go runtime ones. + +Along with these, metrics capsule_proxy_response_time_seconds and capsule_proxy_requests_total have been introduced and are specific to the Capsule Proxy code-base and functionalities. + +capsule_proxy_response_time_seconds offers a bucket representation of the HTTP request duration. The available variables for these metrics are the following ones: + +path: the HTTP path of every single request that Capsule Proxy passes to the upstream +capsule_proxy_requests_total counts the global requests that Capsule Proxy is passing to the upstream with the following labels. + +path: the HTTP path of every single request that Capsule Proxy passes to the upstream +status: the HTTP status code of the request + + diff --git a/content/en/docs/proxy/setup/options.md b/content/en/docs/proxy/setup/options.md new file mode 100644 index 0000000..a957770 --- /dev/null +++ b/content/en/docs/proxy/setup/options.md @@ -0,0 +1,111 @@ +--- +title: Controller Options +description: > + Configure the Capsule Proxy Controller +weight: 2 +--- + +You can customize the Capsule Proxy with the following configurations. + +## Controller Options + +You can provide additional options via the helm chart: + +```yaml +options: + extraArgs: + - --disable-caching=true +``` + +Options are also available as dedicated configuration values: + +```yaml +# Controller Options +options: + # -- Set the listening port of the capsule-proxy + listeningPort: 9001 + # -- Set leader election to true if you are running n-replicas + leaderElection: false + # -- Set the log verbosity of the capsule-proxy with a value from 1 to 10 + logLevel: 4 + # -- Name of the CapsuleConfiguration custom resource used by Capsule, required to identify the user groups + capsuleConfigurationName: default + # -- Define which groups must be ignored while proxying requests + ignoredUserGroups: [] + # -- Specify if capsule-proxy will use SSL + oidcUsernameClaim: preferred_username + # -- Specify if capsule-proxy will use SSL + enableSSL: true + # -- Set the directory, where SSL certificate and keyfile will be located + SSLDirectory: /opt/capsule-proxy + # -- Set the name of SSL certificate file + SSLCertFileName: tls.crt + # -- Set the name of SSL key file + SSLKeyFileName: tls.key + # -- Specify if capsule-proxy will generate self-signed SSL certificates + generateCertificates: false + # -- Specify additional subject alternative names for the self-signed SSL + additionalSANs: [] + # -- Specify an override for the Secret containing the certificate for SSL. Default value is empty and referring to the generated certificate. + certificateVolumeName: "" + # -- Set the role bindings reflector resync period, a local cache to store mappings between users and their namespaces. [Use a lower value in case of flaky etcd server connections.](https://github.com/projectcapsule/capsule-proxy/issues/174) + rolebindingsResyncPeriod: 10h + # -- Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector. + disableCaching: false + # -- Enable the rolebinding reflector, which allows to list the namespaces, where a rolebinding mentions a user. + roleBindingReflector: false + # -- Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate] + authPreferredTypes: "BearerToken,TLSCertificate" + # -- QPS to use for interacting with Kubernetes API Server. + clientConnectionQPS: 20 + # -- Burst to use for interacting with kubernetes API Server. + clientConnectionBurst: 30 + # -- Enable Pprof for profiling + pprof: false +``` + +The following options are available for the Capsule Proxy Controller: + +```shell + --auth-preferred-types string Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate] + First match is used and can be specified multiple times as comma separated values or by using the flag multiple times. (default "[TLSCertificate,BearerToken]") + --capsule-configuration-name string Name of the CapsuleConfiguration used to retrieve the Capsule user groups names (default "default") + --capsule-user-group strings Names of the groups for capsule users (deprecated: use capsule-configuration-name) + --client-connection-burst int32 Burst to use for interacting with kubernetes apiserver. (default 30) + --client-connection-qps float32 QPS to use for interacting with kubernetes apiserver. (default 20) + --disable-caching Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector (default: false) + --enable-leader-election Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager. + --enable-pprof Enables Pprof endpoint for profiling (not recommend in production) + --enable-reflector Enable rolebinding reflector. The reflector allows to list the namespaces, where a rolebinding mentions a user + --enable-ssl Enable the bind on HTTPS for secure communication (default: true) (default true) + --feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are: + AllAlpha=true|false (ALPHA - default=false) + AllBeta=true|false (BETA - default=false) + ProxyAllNamespaced=true|false (ALPHA - default=false) + ProxyClusterScoped=true|false (ALPHA - default=false) + SkipImpersonationReview=true|false (ALPHA - default=false) + --ignored-impersonation-group strings Names of the groups which are not used for impersonation (considered after impersonation-group-regexp) + --ignored-user-group strings Names of the groups which requests must be ignored and proxy-passed to the upstream server + --impersonation-group-regexp string Regular expression to match the groups which are considered for impersonation + --listening-port uint HTTP port the proxy listens to (default: 9001) (default 9001) + --metrics-addr string The address the metric endpoint binds to. (default ":8080") + --oidc-username-claim string The OIDC field name used to identify the user (default: preferred_username) (default "preferred_username") + --rolebindings-resync-period duration Resync period for rolebindings reflector (default 10h0m0s) + --ssl-cert-path string Path to the TLS certificate (default: /opt/capsule-proxy/tls.crt) + --ssl-key-path string Path to the TLS certificate key (default: /opt/capsule-proxy/tls.key) + --webhook-port int The port the webhook server binds to. (default 9443) + --zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) + --zap-encoder encoder Zap log encoding (one of 'json' or 'console') + --zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic'or any integer value > 0 which corresponds to custom debug levels of increasing verbosity + --zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic'). + --zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'. +``` + +## Feature Gates + +Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available: + +| **Feature Gate** | **Default Value** | **Description** | +| :--- | :--- | :--- | +| `SkipImpersonationReview` | `false` | `SkipImpersonationReview` allows to skip the impersonation review for all requests containing impersonation headers (user and groups). **DANGER:** Enabling this flag allows any user to impersonate as any user or group essentially bypassing any authorization. Only use this option in trusted environments where authorization/authentication is offloaded to external systems. | +| `ProxyClusterScoped` | `false` | `ProxyClusterScoped` allows to proxy all clusterScoped objects for all tenant users. These can be defined via [ProxySettings](./proxysettings) | diff --git a/content/en/docs/replications/_index.md b/content/en/docs/replications/_index.md index b34ffdb..d133537 100644 --- a/content/en/docs/replications/_index.md +++ b/content/en/docs/replications/_index.md @@ -5,142 +5,9 @@ description: > Replicate resources across tenants or namespaces --- -When developing an Internal Developer Platform the Platform Administrator could want to propagate a set of resources. These could be Secret, ConfigMap, or other kinds of resources that the tenants would require to use the platform. We provide dedicated Custom Resource Definitions to achieve this goal. Either on [tenant basis](#tenantresource) or [tenant-wide](#globaltenantresource). +Capsule provides two dedicated Custom Resource Definitions for propagating Kubernetes resources across Tenant Namespaces, covering both the cluster administrator and Tenant owner personas: -## GlobalTenantResource +- **[GlobalTenantResource](./global/)** — cluster-scoped, managed by cluster administrators. Selects Tenants by label and replicates resources into all matching Tenant Namespaces. +- **[TenantResource](./tenant/)** — namespace-scoped, managed by Tenant owners. Replicates resources across the Namespaces within a single Tenant. -When developing an Internal Developer Platform the Platform Administrator could want to propagate a set of resources. These could be Secret, ConfigMap, or other kinds of resources that the tenants would require to use the platform. - - > A generic example could be the container registry secrets, especially in the context where the Tenants can just use a specific registry. - -Starting from Capsule v0.2.0, a new set of Custom Resource Definitions have been introduced, such as the GlobalTenantResource, let's start with a potential use-case using the personas described at the beginning of this document. - -Bill created the Tenants for Alice using the Tenant CRD, and labels these resources using the following command: - -```bash -$ kubectl label tnt/solar energy=renewable -tenant solar labeled - -$ kubectl label tnt/green energy=renewable -tenant green labeled -``` - -In the said scenario, these Tenants must use container images from a trusted registry, and that would require the usage of specific credentials for the image pull. - -The said container registry is deployed in the cluster in the namespace harbor-system, and this Namespace contains all image pull secret for each Tenant, e.g.: a secret named `harbor-system/fossil-pull-secret` as follows. - -```bash -$ kubectl -n harbor-system get secret --show-labels -NAME TYPE DATA AGE LABELS -renewable-pull-secret Opaque 1 28s tenant=renewable -``` - -These credentials would be distributed to the Tenant owners manually, or vice-versa, the owners would require those. Such a scenario would be against the concept of the self-service solution offered by Capsule, and Bill can solve this by creating the `GlobalTenantResource` as follows. - -```yaml -apiVersion: capsule.clastix.io/v1beta2 -kind: GlobalTenantResource -metadata: - name: renewable-pull-secrets -spec: - tenantSelector: - matchLabels: - energy: renewable - resyncPeriod: 60s - resources: - - namespacedItems: - - apiVersion: v1 - kind: Secret - namespace: harbor-system - selector: - matchLabels: - tenant: renewable -``` - -The GlobalTenantResource is a cluster-scoped resource, thus it has been designed for cluster administrators and cannot be used by Tenant owners: for that purpose, the [TenantResource](#tenantresource) one can help. - -> Capsule will select all the Tenant resources according to the key tenantSelector. Each object defined in the namespacedItems and matching the provided selector will be replicated into each Namespace bounded to the selected Tenants. Capsule will check every 60 seconds if the resources are replicated and in sync, as defined in the key resyncPeriod. - -## Scope - -You can change to scope - - -## TenantResource - -Although Capsule is supporting a few amounts of personas, it can be used to allow building an Internal Developer Platform used barely by [Tenant owners](/docs/tenants/permissions#ownership), or users created by these thanks to Service Account. - -In a such scenario, a Tenant Owner would like to distribute resources across all the Namespace of their Tenant, without the need to establish a manual procedure, or the need for writing a custom automation. - -The Namespaced-scope API TenantResource allows to replicate resources across the Tenant's Namespace. - -The Tenant owners must have proper RBAC configured in order to create, get, update, and delete their TenantResource CRD instances. This can be achieved using the Tenant key additionalRoleBindings or a custom Tenant owner role, compared to the default one (admin). You can for example create this clusterrole, which will aggregate to the admin role, to allow the Tenant Owner to create TenantResource objects. This allows all users with the rolebinding to `admin` to create TenantResource objects. - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: allow-tenant-resources - labels: - rbac.authorization.k8s.io/aggregate-to-admin: "true" -rules: -- apiGroups: ["capsule.clastix.io"] - resources: ["tenantresources"] - verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] -``` - -For our example, Alice, the project lead for the solar tenant, wants to provision automatically a DataBase resource for each Namespace of their Tenant: these are the Namespace list. - -```bash -$ kubectl get namespaces -l capsule.clastix.io/tenant=solar --show-labels -NAME STATUS AGE LABELS -solar-1 Active 59s capsule.clastix.io/tenant=solar,environment=production,kubernetes.io/metadata.name=solar-1,name=solar-1 -solar-2 Active 58s capsule.clastix.io/tenant=solar,environment=production,kubernetes.io/metadata.name=solar-2,name=solar-2 -solar-system Active 62s capsule.clastix.io/tenant=solar,kubernetes.io/metadata.name=solar-system,name=solar-system -``` - -Alice creates a TenantResource in the Tenant namespace solar-system as follows. - -```yaml -apiVersion: capsule.clastix.io/v1beta2 -kind: TenantResource -metadata: - name: solar-db - namespace: solar-system -spec: - resyncPeriod: 60s - resources: - - additionalMetadata: - labels: - "replicated-by": "capsule" - namespaceSelector: - matchLabels: - environment: production - rawItems: - - apiVersion: postgresql.cnpg.io/v1 - kind: Cluster - metadata: - name: "postgres-{{namespace}}" - spec: - description: PostgreSQL cluster for the {{tenant.name}} Project - instances: 3 - postgresql: - pg_hba: - - hostssl app all all cert - primaryUpdateStrategy: unsupervised - storage: - size: 1Gi -``` - -The expected result will be the object Cluster for the API version postgresql.cnpg.io/v1 to get created in all the Solar tenant namespaces matching the label selector declared by the key `namespaceSelector`. - -```bash -$ kubectl get clusters.postgresql.cnpg.io -A -NAMESPACE NAME AGE INSTANCES READY STATUS PRIMARY -solar-1 postgres-solar-1 80s 3 3 Cluster in healthy state postgresql-1 -solar-2 postgres-solar-2 80s 3 3 Cluster in healthy state postgresql-1 -``` - -The TenantResource object has been created in the namespace `solar-system` that doesn't satisfy the Namespace selector. Furthermore, Capsule will automatically inject the required labels to avoid a `TenantResource` could start polluting other Namespaces. - -Eventually, using the key namespacedItem, it is possible to reference existing objects to get propagated across the other Tenant namespaces: in this case, a Tenant Owner can just refer to objects in their Namespaces, preventing a possible escalation referring to non owned objects. +Both CRDs follow the same structure: resources are defined in `spec.resources` blocks, reconciled on a configurable `resyncPeriod`, and support [Go-template-based generators](/docs/operating/templating/) for dynamic resource creation. diff --git a/content/en/docs/replications/global.md b/content/en/docs/replications/global.md new file mode 100644 index 0000000..00f28ca --- /dev/null +++ b/content/en/docs/replications/global.md @@ -0,0 +1,1410 @@ +--- +title: GlobalTenantResources +weight: 1 +description: > + Replicate resources across tenants or namespaces as Cluster Administrator. +--- + +## Overview + +`GlobalTenantResource` is a cluster-scoped CRD designed for cluster administrators. It lets you automatically replicate Kubernetes resources - such as Secrets, ConfigMaps, or custom resources - into the Namespaces of selected Tenants. Tenant owners cannot create `GlobalTenantResource` objects; for tenant-scoped replication, see [TenantResource](../tenant/). + +The diagram below shows that an Administrator can create a `GlobalTenantResource`. In the `GlobalTenantResource` spec, an Administrator specifies which resource they would like to replicate, and where this resource should be replicated to. When applied, this resource gets automatically distributed across all Namespaces of the `Tenants` that are selected in the `GlobalTenantResource`. + +![Global Tenant Resource Replication overview](/images/content/replication-globaltenantresource.png) + +A common use case is distributing image pull secrets to all Tenants that must use a specific container registry. In the following example, Bill labels two Tenants and then creates a `GlobalTenantResource` to push the corresponding pull secret into each of their Namespaces automatically. + +```bash +$ kubectl label tnt/solar energy=renewable +tenant solar labeled + +$ kubectl label tnt/green energy=renewable +tenant green labeled +``` + +The pull secret already exists in the `harbor-system` namespace, labelled accordingly: + +```bash +$ kubectl -n harbor-system get secret --show-labels +NAME TYPE DATA AGE LABELS +imagePullSecret Opaque 1 28s tenant=renewable +``` + +Without automation, these credentials would need to be distributed manually - against the self-service principle of Capsule. Bill solves this with a single `GlobalTenantResource`: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: renewable-pull-secrets +spec: + tenantSelector: + matchLabels: + energy: renewable + resyncPeriod: 60s + resources: + - namespacedItems: + - apiVersion: v1 + kind: Secret + namespace: harbor-system + selector: + matchLabels: + tenant: renewable +``` + +Capsule selects all Tenants matching `tenantSelector`, then replicates every item in `namespacedItems` into each Namespace belonging to those Tenants. The controller reconciles on the interval defined by `resyncPeriod`. + +> Objects managed by this controller can be either **created** (new objects) or **adopted** (existing objects). See [Object Management](#object-management) in the Advanced section for full details. + +--- + +## Basic Usage + +### TenantSelector + +A block that describes which Tenants the resource should be replicated to. `matchLabels` and `matchExpressions` can be used to select the desired Tenants. To select all tenants with the label `energy: renewable`, use: + +```yaml + tenantSelector: + matchLabels: + energy: renewable +``` + +TenantSelector is an optional field. If not set, the resources will be replicated to all tenants. + +### Resources + +A resource block defines *what* to replicate. Multiple blocks can be stacked in the `resources` array, each using one or more of the strategies below. + +#### NamespaceSelector + +The `namespaceSelector` field restricts replication to Namespaces matching a label selector. Capsule also protects selected resources from modification by Tenant users via its webhook. + +#### AdditionalMetadata + +Use `additionalMetadata` to attach extra `labels` and `annotations` to every generated object. [Fast Template values](/docs/operating/templating/#fast-templates) are supported: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-cluster-rbac +spec: + scope: Tenant + resources: + - additionalMetadata: + labels: + k8s.company.com/tenant: "{{tenant.name}}" + annotations: + k8s.company.com/cost-center: "inv-120" + generators: + - missingKey: error + template: | + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: tenant:{{$.tenant.metadata.name}}:priority + labels: + k8s.company.com/tenant: "test" + rules: + - apiGroups: ["scheduling.k8s.io"] + verbs: ["get"] + resources: ["priorityclasses"] +``` + +When the same label key appears in both `additionalMetadata` and the template, `additionalMetadata` takes priority. + +The following labels are always stripped because they are reserved for the controller: + + * `capsule.clastix.io/resources` + * `projectcapsule.dev/created-by` + * `capsule.clastix.io/managed-by` + * `projectcapsule.dev/managed-by` + +#### NamespacedItems + +Reference existing resources for replication across Tenant Namespaces. The controller validates that any resource kind listed here is namespace-scoped; cluster-scoped kinds are rejected with an error. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + + # Replicate all Configmaps labeled with projectcapsule.dev/replicate: "true" + - apiVersion: v1 + kind: ConfigMap + selector: + matchLabels: + projectcapsule.dev/replicate: "true" + + # Replicate all Configmaps labeled with projectcapsule.dev/replicate: "true" and in namespace capsule-system + - apiVersion: v1 + kind: ConfigMap + namespace: capsule-system + selector: + matchLabels: + projectcapsule.dev/replicate: "true" + + # Replicate Configmap named "logging-config" in namespace capsule-system labeled with projectcapsule.dev/replicate: "true" and in namespace capsule-system + - apiVersion: v1 + kind: ConfigMap + name: logging-config + namespace: capsule-system +``` + +**Note**: Resources with the label `projectcapsule.dev/created-by: resources` are ignored by `namespacedItems` to prevent reconciliation loops. + +If you try to define a cluster-scoped resource under `namespacedItems`, the reconciliation will fail immediately: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + - namespacedItems: + - apiVersion: addons.projectcapsule.dev/v1alpha1 + kind: SopsProvider + name: infrastructure-provider + optional: true + +status: + conditions: + - lastTransitionTime: "2026-01-15T21:04:15Z" + message: cluster-scoped kind addons.projectcapsule.dev/v1alpha1/SopsProvider is + not allowed + reason: Failed + status: "False" + type: Ready +``` + +##### Name + +Providing `name` triggers a `GET` request for that single resource rather than a `LIST`. You must also specify `namespace` when using `name` in a `GlobalTenantResource`: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-resource-replications +spec: + resources: + - namespacedItems: + - apiVersion: v1 + kind: ConfigMap + name: config-namespace + optional: true +status: + conditions: + - lastTransitionTime: "2026-01-15T21:10:17Z" + message: 'failed to get ConfigMap/config-namespace: an empty namespace may not + be set when a resource name is provided' + reason: Failed + status: "False" + type: Ready +``` + +##### Namespace + +Providing only `namespace` performs a `LIST` of all resources of that kind in that namespace: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + # Fetches all configmaps in the namespace tenants-system + - apiVersion: v1 + kind: ConfigMap + namespace: "tenants-system" + + # Fetches specific configmaps matching the selector in the namespaces tenants-system + - apiVersion: v1 + kind: ConfigMap + namespace: "tenants-system" + selector: + matchLabels: + projectcapsule.dev/replicate: "true" +``` + +[Fast Templates](/docs/operating/templating/#fast-templates) are supported for `namespace`: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + # Fetch ConfigMaps labeled with the tenant name and replicate them into each Tenant Namespace + - apiVersion: v1 + kind: Secret + namespace: "{{tenant.name}}-system" +``` + +**Note**: When using `TenantResource` instead of `GlobalTenantResource`, the `namespace` field has no effect - resources can only be referenced from the Namespace where the `TenantResource` object was created. + +##### Selector + +When using `selector`, the selector labels are stripped from the replicated objects. This prevents the replicated copy from also matching the source selector, which would cause a circular reconciliation loop. + +Source `ConfigMap`: + +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: app-config + labels: + projectcapsule.dev/replicate: "true" + namespace: wind-test +data: + player_initial_lives: "3" + ui_properties_file_name: "user-interface.properties" +``` + +`TenantResource`: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: app-config +spec: + resources: + - namespacedItems: + - apiVersion: v1 + kind: ConfigMap + selector: + matchLabels: + projectcapsule.dev/replicate: "true" +``` + +Resulting object in `wind-prod` (notice the absence of `projectcapsule.dev/replicate`): + +```yaml +apiVersion: v1 +data: + player_initial_lives: "3" + ui_properties_file_name: "user-interface.properties" +kind: ConfigMap +metadata: + labels: + projectcapsule.dev/created-by: resources + projectcapsule.dev/managed-by: resources + name: app-config + namespace: wind-prod + resourceVersion: "784529" + uid: 5f10a3f3-863e-4f45-9454-cff8f5bce86a +``` + +[Fast Templates](/docs/operating/templating/#fast-templates) are supported for `selector`: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + # Fetch ConfigMaps labeled with the tenant name and replicate them into each Tenant Namespace + - apiVersion: v1 + kind: ConfigMap + selector: + matchLabels: + company.com/replicate-for: "{{tenant.name}}" +``` + +#### Raw + +Raw items let you define resources inline as standard Kubernetes manifests. Use this when the resource does not yet exist in the cluster, or when you want to define it directly in the spec. [Fast Templates](/docs/operating/templating/#fast-templates) are supported. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 300s + resources: + - rawItems: + - apiVersion: v1 + kind: LimitRange + metadata: + name: "{{tenant.name}}-{{namespace}}-resource-constraint" + spec: + limits: + - default: # this section defines default limits + cpu: 500m + defaultRequest: # this section defines default requests + cpu: 500m + max: # max and min define the limit range + cpu: "1" + min: + cpu: 100m + type: Container +``` + +The following example creates a [`SopsProvider`](https://github.com/peak-scale/sops-operator) for each Tenant: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-sops-providers +spec: + resyncPeriod: 600s + scope: Tenant + resources: + - rawItems: + - apiVersion: addons.projectcapsule.dev/v1alpha1 + kind: SopsProvider + metadata: + name: "{{tenant.name}}-secrets" + spec: + keys: + - namespaceSelector: + matchLabels: + capsule.clastix.io/tenant: "{{tenant.name}}" + sops: + - namespaceSelector: + matchLabels: + capsule.clastix.io/tenant: "{{tenant.name}}" +``` + +Because [Server-Side Apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/) is used, you only need to specify the fields you want to manage - the full resource spec is not required. + +For more advanced templating, consider [Generators](#generators). + +#### Generators + +Generators render one or more Kubernetes objects from a Go template string. The template content must be valid YAML; multi-document output separated by `---` is supported. The template engine is based on [go-sprout](https://github.com/go-sprout/sprout) - see [available functions](/docs/operating/templating/#sprout-templating). + +A simple example that creates a `ClusterRole` per Tenant: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-cluster-rbac +spec: + scope: Tenant + resources: + - generators: + - missingKey: error + template: | + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: tenant:{{$.tenant.metadata.name}}:reader + rules: + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "watch", "list"] +``` + +Templates can also produce multiple objects using flow control: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-priority-rbac +spec: + scope: Tenant + resources: + - generators: + - missingKey: error + template: | + {{- range $.tenant.status.classes.priority }} + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: tenant:{{$.tenant.metadata.name}}:priority:{{.}} + rules: + - apiGroups: ["scheduling.k8s.io"] + resources: ["priorityclasses"] + resourceNames: ["{{.}}"] + verbs: ["get"] + {{- end }} +``` + +See [Base Context](#base-context) for available template variables. To load additional resources into the template context, see [Context](#context) in the Advanced section. + +##### Template Snippets + +Some snippets that might be useful for certain cases. + +###### Names + +Extract the `Tenant` name: + +```html +{{ $.tenant.metadata.name }} +``` + +Extract the `Namespace` name: + +```html +{{ $.namespace.metadata.name }} +``` + +###### Foreach Owner + +Iterate over all owners of a Tenant: + +```html + {{- range $.tenant.status.owners }} + {{ .kind }}: {{ .name }} + {{- end }} +``` + +##### MissingKey + +Controls template behaviour when a referenced context key is absent. + +###### Invalid + +Continues execution silently. Missing keys render as the string `""`. + +This definition with the missing context: + +```yaml +kind: GlobalTenantResource +metadata: + name: missing-key +spec: + resources: + - generators: + - missingKey: invalid + template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: show-key + data: + value: {{ $.custom.account.name }} +``` + +Turns into after templating: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: show-key +data: + value: "" +``` + +###### Zero + +**This is the default behavior.** Missing keys resolve to the zero value of their type (usually an empty string). + +This definition with the missing context: + +```yaml +kind: GlobalTenantResource +metadata: + name: missing-key +spec: + resources: + - generators: + - missingKey: zero + template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: show-key + data: + value: {{ $.custom.account.name }} +``` + +Turns into after templating: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: show-key +data: + value: "" +``` + +###### Error + +Stops execution immediately with an error when a required key is missing. + +This definition with the missing context: + +```yaml +kind: GlobalTenantResource +metadata: + name: missing-key +spec: + resources: + - generators: + - missingKey: error + template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: show-key + data: + value: {{ $.custom.account.name }} +``` + +Will error the `GlobalTenantResources`: + +```shell +NAME ITEMS READY STATUS AGE +missing-key 6 False error running generator: template: tpl:8:7: executing "tpl" at <$.namespace.name>: map has no entry for key "name" 9m5s +``` + +--- + +### Reconciliation + +#### Period + +`GlobalTenantResources` reconcile on the interval defined by `resyncPeriod`. The default is `60s`. Capsule does not watch source resources for changes; it reconciles periodically. A very short interval on large clusters with many Tenants and Namespaces can cause performance issues - tune this value accordingly. + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: renewable-pull-secrets +spec: + resyncPeriod: 300s # 5 minutes + resources: + - namespacedItems: + - apiVersion: v1 + kind: Secret + namespace: harbor-system + selector: + matchLabels: + tenant: renewable +``` + +#### Manual + +To trigger an immediate reconciliation, add the `reconcile.projectcapsule.dev/requestedAt` annotation. The annotation is removed once reconciliation completes, making the process repeatable. + +```bash +kubectl annotate globaltenantresource renewable-pull-secrets \ + reconcile.projectcapsule.dev/requestedAt="$(date -Iseconds)" +``` + +--- + +### Scope + +By default, a `GlobalTenantResource` replicates resources into **every Namespace** of the selected Tenants. Setting `scope: Tenant` changes this to replicate once per Tenant instead. + +Possible values: + + * `Tenant`: Replicate once per Tenant. + * `Namespace`: Replicate into each Namespace of the selected Tenants. *(Default)* + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-sops-providers +spec: + resyncPeriod: 60s + scope: Tenant + resources: + - rawItems: + - apiVersion: addons.projectcapsule.dev/v1alpha1 + kind: SopsProvider + metadata: + name: {{tenant.name}}-secrets + spec: + keys: + - namespaceSelector: + matchLabels: + capsule.clastix.io/tenant: {{tenant.name}} + sops: + - namespaceSelector: + matchLabels: + capsule.clastix.io/tenant: {{tenant.name}} +``` + +Using the `scope: Tenant` is mainly useful when you want to deploy a cluster-scoped resource once per tenant, such as the `SopsProvider` above. + +**Note:** When `scope: Tenant` is set, `namespacedItems` entries are not processed, since there is no target Namespace in that scope. + +--- + +### Impersonation + +{{% alert title="Information" color="warning" %}} +Without a configured ServiceAccount, the Capsule controller ServiceAccount is used for replication operations. This may allow privilege escalation if the controller has broader permissions than Tenant owners. +{{% /alert %}} + +Enabling impersonation ensures that replication operations run under a specific ServiceAccount identity, providing a proper audit trail and limiting privilege exposure. You can check which ServiceAccount is currently in use via the object's status: + +```bash +kubectl get globaltenantresource custom-cm -o jsonpath='{.status.serviceAccount}' | jq +{ + "name": "capsule", + "namespace": "capsule-system" +} +``` + +To use a different ServiceAccount, set the `serviceAccount` field on the object: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-resource-replications +spec: + serviceAccount: + name: "default" + namespace: "kube-system" + resources: + - namespacedItems: + - apiVersion: v1 + kind: ConfigMap + name: "config-namespace" +``` + +If the ServiceAccount lacks the required RBAC, replication will fail with a permission error: + +```yaml + - kind: ConfigMap + name: game-demo + namespace: wind-prod + status: + created: true + message: 'apply failed for item 0/raw-0: applying object failed: configmaps + "game-demo" is forbidden: User "system:serviceaccount:kube-system:default" + cannot patch resource "configmaps" in API group "" in the namespace "wind-prod"' + status: "False" + type: Ready + tenant: wind + version: v1 +``` + +Grant the ServiceAccount the necessary permissions: + +```yaml +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: capsule-tenant-replications +rules: +- apiGroups: [""] + resources: ["configmaps"] + verbs: ["list", "get", "patch", "create", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: capsule-tenant-replications +subjects: +- kind: ServiceAccount + name: default + namespace: kube-system +roleRef: + kind: ClusterRole + name: capsule-tenant-replications + apiGroup: rbac.authorization.k8s.io + +``` + +#### Required Permissions + +The following permissions are required for each resource type managed by the replication feature: + + * `get` (always required) + * `create` (always required) + * `patch` (always required) + * `delete` (always required) + * `list` (required for [Namespaced Items](#namespaceditems) and [Context](#context)) + +Missing any of these will cause replication to fail. + +#### Default ServiceAccount + +To ensure all `GlobalTenantResource` objects use a controlled identity by default, configure a cluster-wide default ServiceAccount in the Capsule manager options. Per-object `serviceAccount` fields override this default. + +[Read more about Impersonation](/docs/operating/setup/configuration/#impersonation). You must provide both the name and namespace of the ServiceAccount: + +```yaml +manager: + options: + impersonation: + globalDefaultServiceAccount: "capsule-default-global" + globalDefaultServiceAccountNamespace: "capsule-system" +``` + +The default ServiceAccount must have sufficient RBAC. The following example allows it to manage Secrets and LimitRanges across all Tenants: + +```yaml +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: capsule-default-global +rules: +- apiGroups: [""] + resources: ["limitranges", "secrets"] + verbs: ["get", "patch", "create", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: capsule-default-global +subjects: +- kind: ServiceAccount + name: capsule-default-global + namespace: capsule-system +roleRef: + kind: ClusterRole + name: capsule-default-global + apiGroup: rbac.authorization.k8s.io +``` + +If a `GlobalTenantResource` attempts to manage a resource type not covered by the default ServiceAccount's ClusterRole, replication will fail with a permissions error: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: default-sa-replication +spec: + resyncPeriod: 60s + resources: + - rawItems: + - apiVersion: v1 + kind: ConfigMap + metadata: + name: game-demo + data: + player_initial_lives: "3" + ui_properties_file_name: "user-interface.properties" +``` + +--- + +## Advanced + +This section covers more advanced features of the Replication setup. + +### Object Management + +Capsule uses [Server-Side Apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/) for all replication operations. Two management modes exist depending on whether the object already existed before reconciliation. + +#### Create + +An object is *Created* when the `GlobalTenantResource` first encounters it - it did not exist prior to reconciliation. Created objects receive the following metadata: + + * `metadata.labels.projectcapsule.dev/created-by`: `resources` + * `metadata.labels.projectcapsule.dev/managed-by`: `resources` + * `metadata.ownerReferences`: Owner reference to the corresponding `GlobalTenantResource` + +```yaml +kind: ConfigMap +metadata: + labels: + projectcapsule.dev/created-by: resources + projectcapsule.dev/managed-by: resources + name: common-config + namespace: green-test + ownerReferences: + - apiVersion: capsule.clastix.io/v1beta2 + kind: GlobalTenantResource + name: tenant-cm-providers + uid: 903395eb-9314-462d-ae19-7c87d71e890b + resourceVersion: "549517" + uid: 23abbb7a-2926-416a-bc72-9f793ebf6080 +``` + +Since we are using [Server-Side Apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/) we can also allow different items making changes to the same object, when it was created by a `GlobalTenantResource`, as long as there are no conflicts: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-cm-registration +spec: + scope: Tenant + resources: + - generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: common-config + namespace: default + data: + {{ $.tenant.metadata.name }}.conf: | + {{ toYAML $.tenant.metadata | nindent 4 }} +``` + +Will result in the following object: + +```yaml +apiVersion: v1 +data: + green.conf: "\ncreationTimestamp: \"2026-02-05T08:03:25Z\"\ngeneration: 2\nlabels:\n + \ customer: a\n kubernetes.io/metadata.name: green\nname: green\nresourceVersion: + \"549455\"\nuid: 7b756efd-cdad-484b-a41f-d1a00d401781 \n" + solar.conf: "\ncreationTimestamp: \"2026-02-05T08:03:25Z\"\ngeneration: 2\nlabels:\n + \ customer: a\n kubernetes.io/metadata.name: solar\nname: solar\nresourceVersion: + \"549521\"\nuid: c2b21703-2321-4789-af9f-65e541c883d5 \n" + wind.conf: "\ncreationTimestamp: \"2026-02-05T13:43:22Z\"\ngeneration: 1\nlabels:\n + \ kubernetes.io/metadata.name: wind\nname: wind\nresourceVersion: \"542629\"\nuid: + 72388253-ff5c-4614-94a2-2fd8cd7cf813 \n" +kind: ConfigMap +metadata: + creationTimestamp: "2026-02-05T15:37:09Z" + labels: + projectcapsule.dev/created-by: resources + projectcapsule.dev/managed-by: resources + name: common-config + namespace: default + ownerReferences: + - apiVersion: capsule.clastix.io/v1beta2 + kind: GlobalTenantResource + name: tenant-sops-providers + uid: 7cf01d19-0555-490f-bd01-a5beff0cbc64 + resourceVersion: "561707" + uid: 33cfe1c6-1c9e-4417-9dd5-26ac0ba3bc85 +``` + +This also works across different `GlobalTenantResources`: + +```yaml +apiVersion: v1 +data: + common.conf: "\ncreationTimestamp: \"2026-02-05T08:03:25Z\"\ngeneration: 2\nlabels:\n + \ customer: a\n kubernetes.io/metadata.name: green\nname: green\nresourceVersion: + \"549455\"\nuid: 7b756efd-cdad-484b-a41f-d1a00d401781 \n" + green.conf: "\ncreationTimestamp: \"2026-02-05T08:03:25Z\"\ngeneration: 2\nlabels:\n + \ customer: a\n kubernetes.io/metadata.name: green\nname: green\nresourceVersion: + \"549455\"\nuid: 7b756efd-cdad-484b-a41f-d1a00d401781 \n" + solar.conf: "\ncreationTimestamp: \"2026-02-05T08:03:25Z\"\ngeneration: 2\nlabels:\n + \ customer: a\n kubernetes.io/metadata.name: solar\nname: solar\nresourceVersion: + \"549521\"\nuid: c2b21703-2321-4789-af9f-65e541c883d5 \n" + wind.conf: "\ncreationTimestamp: \"2026-02-05T13:43:22Z\"\ngeneration: 1\nlabels:\n + \ kubernetes.io/metadata.name: wind\nname: wind\nresourceVersion: \"542629\"\nuid: + 72388253-ff5c-4614-94a2-2fd8cd7cf813 \n" +kind: ConfigMap +metadata: + creationTimestamp: "2026-02-05T15:37:09Z" + labels: + projectcapsule.dev/created-by: resources + projectcapsule.dev/managed-by: resources + name: common-config + namespace: default + ownerReferences: + - apiVersion: capsule.clastix.io/v1beta2 + kind: GlobalTenantResource + name: tenant-sops-providers + uid: 7cf01d19-0555-490f-bd01-a5beff0cbc64 + - apiVersion: capsule.clastix.io/v1beta2 + kind: GlobalTenantResource + name: tenant-cm-registration + uid: b2d34727-b403-4e2a-9115-232ba61d3c69 + resourceVersion: "562881" + uid: 33cfe1c6-1c9e-4417-9dd5-26ac0ba3bc85 +``` + + However, when try to manage the same field, we will get an error: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-cm-registration +spec: + scope: Tenant + resources: + generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: common-config + namespace: default + data: + common.conf: | + {{ toYAML $.tenant.metadata.name | nindent 4 }} +``` + +We can see a Conflict Error in the `GlobalTenantResource` status: + +```yaml +kubectl get globaltenantresource tenant-cm-registration -o yaml + +... + + status: + processedItems: + - kind: ConfigMap + name: common-config + namespace: default + status: + lastApply: "2026-02-05T15:52:26Z" + status: "True" + type: Ready + tenant: wind + version: v1 + - kind: ConfigMap + name: common-config + namespace: default + status: + created: true + message: 'apply failed for item 0/generator-0-0: applying object failed: Apply + failed with 1 conflict: conflict with "projectcapsule.dev/resource/cluster/tenant-cm-registration//default/wind/": + .data.common.conf' + status: "False" + type: Ready + tenant: green + version: v1 + - kind: ConfigMap + name: common-config + namespace: default + status: + created: true + message: 'apply failed for item 0/generator-0-0: applying object failed: Apply + failed with 1 conflict: conflict with "projectcapsule.dev/resource/cluster/tenant-cm-registration//default/wind/": + .data.common.conf' + status: "False" + type: Ready + tenant: solar + version: v1 +``` + +You can check the `created` property on each item's status to determine whether it was created or adopted. Field conflicts can be resolved with [Force](#force). + +##### Pruning + +When pruning is enabled, *Created* objects are deleted when they fall out of scope. When pruning is disabled, the following metadata is removed instead: + + * `metadata.labels.projectcapsule.dev/managed-by`: `resources` + * `metadata.ownerReferences`: owner reference to the `GlobalTenantResource` + +The label `metadata.labels.projectcapsule.dev/created-by` is preserved after pruning, allowing another `GlobalTenantResource` or `TenantResource` to take ownership without explicit adoption. To prevent re-adoption, remove or change this label manually. + +#### Adopt + +By default, a `GlobalTenantResource` cannot modify objects it did not create. Adoption must be explicitly enabled. Adopted objects receive the following metadata: + + * `metadata.labels.projectcapsule.dev/managed-by`: `resources` + +For example the following `GlobalTenantResource` tries to change the content of the existing `argo-rbac` `ConfigMap`: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: argo-cd-permission +spec: + resources: + - generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: argocd-rbac-cm + data: + {{ $.tenant.metadata.name }}.csv: | + {{- range $.tenant.status.owners }} + p, {{ .name }}, applications, sync, my-{{ $.tenant.metadata.name }}/*, allow + {{- end }} +``` + +We can see, that we get an error for all items. Telling us, we can not overwrite an existing object: + +```yaml +kubectl get globaltenantresource argo-cd-permission -o yaml + +... + processedItems: + - kind: ConfigMap + name: argocd-rbac-cm + namespace: argocd + status: + message: 'apply failed for item 0/generator-0-0: resource evaluation: resource + v1/ConfigMap argocd/argocd-rbac-cm exists and cannot be adopted' + status: "False" + type: Ready + tenant: green + version: v1 + - kind: ConfigMap + name: argocd-rbac-cm + namespace: argocd + status: + message: 'apply failed for item 0/generator-0-0: resource evaluation: resource + v1/ConfigMap argocd/argocd-rbac-cm exists and cannot be adopted' + status: "False" + type: Ready + tenant: solar + version: v1 + - kind: ConfigMap + name: argocd-rbac-cm + namespace: argocd + status: + message: 'apply failed for item 0/generator-0-0: resource evaluation: resource + v1/ConfigMap argocd/argocd-rbac-cm exists and cannot be adopted' + status: "False" + type: Ready + tenant: wind + version: v1 +``` + +If we want to allow that, we can set the `adopt` property to `true`: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: argo-cd-permission +spec: + settings: + adopt: true + resources: + - generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: argocd-rbac-cm + data: + {{ $.tenant.metadata.name }}.csv: | + {{- range $.tenant.status.owners }} + p, {{ .name }}, applications, sync, {{ $.tenant.metadata.name }}/*, allow + {{- end }} +``` + +When adoption is enabled, resources can be modified. Note that if multiple operators manage the same resource, all must use Server-Side Apply to avoid conflicts. + +```shell +kubectl get cm -n argocd argocd-rbac-cm -o yaml +apiVersion: v1 +data: + policy.csv: | + p, my-org:team-alpha, applications, sync, my-project/*, allow + g, my-org:team-beta, role:admin + g, user@example.org, role:admin + g, admin, role:admin + g, role:admin, role:readonly + policy.default: role:readonly + scopes: '[groups, email]' + + green.csv: |2 + + p, oidc:org:devops, applications, sync, green/*, allow + p, bob, applications, sync, green/*, allow + solar.csv: |2 + + p, oidc:org:platform, applications, sync, solar/*, allow + p, alice, applications, sync, solar/*, allow + wind.csv: |2 + + p, oidc:org:devops, applications, sync, wind/*, allow + p, joe, applications, sync, wind/*, allow +kind: ConfigMap +``` + +##### Pruning + +When pruning is enabled, adoption is reverted - the patches introduced by the `GlobalTenantResource` are removed from the object. When pruning is disabled, only the `metadata.labels.projectcapsule.dev/managed-by` label is removed. + +--- + +### DependsOn + +A `GlobalTenantResource` can declare dependencies on other `GlobalTenantResource` objects using `dependsOn`. The controller will not reconcile the resource until all declared dependencies are in `Ready` state. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: gitops-owners +spec: + resyncPeriod: 60s + dependsOn: + - name: custom-cm + resources: + - additionalMetadata: + labels: + projectcapsule.dev/tenant: "{{tenant.name}}" + rawItems: + - apiVersion: capsule.clastix.io/v1beta2 + kind: TenantOwner + metadata: + name: "{{tenant.name}}-{{namespace}}" + spec: + clusterRoles: + - capsule-namespace-deleter + - admin + kind: ServiceAccount + name: "system:serviceaccount:{{namespace}}:gitops-reconciler" +``` + +We can observe the status of the `GlobalTenantResource` reflecting, that it depends `GlobalTenantResource` is not yet ready. + +```bash +kubectl get globaltenantresource + +NAME ITEM COUNT READY STATUS AGE +custom-cm 6 False applying of 6 resources failed 12h +gitops-owners 6 False dependency custom-cm-2 not found 8h +``` + +If a dependency does not exist, we can observe a similar status message when describing the `GlobalTenantResource` object. + +```bash +kubectl get globaltenantresource gitops-owners + +NAME ITEM COUNT READY STATUS AGE +gitops-owners 6 False dependency custom-cm-2 not found 8h +``` + +Dependencies are evaluated in the order they are declared in the `dependsOn` array. + +--- + +### Force + +Setting `settings.force: true` instructs Capsule to [force-apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts) changes on Server-Side Apply conflicts, claiming field ownership even if another manager already holds it. + +**This option should generally be avoided.** Forcing ownership over a field managed by another operator will almost certainly cause a reconcile war. Only use it in scenarios where you intentionally want Capsule to win ownership disputes. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-technical-accounts +spec: + settings: + force: true + resources: + - generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: shared-config + data: + common.conf: | + {{ toYAML $.tenant.metadata | nindent 4 }} +``` + +--- + +### Context + +The `context` field lets you load additional Kubernetes resources into the template rendering context. This is useful when you need to iterate over existing objects as part of your template logic: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-sops-providers +spec: + resyncPeriod: 600s + resources: + - context: + resources: + - index: secrets + apiVersion: v1 + kind: Secret + namespace: "{{.namespace}}" + selector: + matchLabels: + pullsecret.company.com: "true" + - index: sa + apiVersion: v1 + kind: ServiceAccount + namespace: "{{.namespace}}" + + generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: show-context + data: + context.yaml: | + {{- toYAML $ | nindent 4 }} +``` + + + + + +#### Base Context + +The following context is always available in generator templates. The `tenant` key is always present. The `namespace` key is only available when the scope is `Namespace` (the default); it is absent when `scope: Tenant` is set. + +```yaml +tenant: + apiVersion: capsule.clastix.io/v1beta2 + kind: Tenant + metadata: + creationTimestamp: "2026-02-06T09:54:30Z" + generation: 1 + labels: + kubernetes.io/metadata.name: wind + name: wind + resourceVersion: "4038" + uid: 93992a2b-cba4-4d33-9d09-da8fc0bfe93c + spec: + additionalRoleBindings: + - clusterRoleName: view + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: wind-users + owners: + - clusterRoles: + - admin + - capsule-namespace-deleter + kind: User + name: joe + permissions: + matchOwners: + - matchLabels: + team: devops + - matchLabels: + tenant: wind + status: + classes: + priority: + - system-cluster-critical + - system-node-critical + storage: + - standard + conditions: + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: reconciled + reason: Succeeded + status: "True" + type: Ready + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: not cordoned + reason: Active + status: "False" + type: Cordoned + namespaces: + - wind-prod + - wind-test + owners: + - clusterRoles: + - admin + - capsule-namespace-deleter + kind: Group + name: oidc:org:devops + - clusterRoles: + - admin + - capsule-namespace-deleter + kind: User + name: joe + size: 2 + spaces: + - conditions: + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: reconciled + reason: Succeeded + status: "True" + type: Ready + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: not cordoned + reason: Active + status: "False" + type: Cordoned + metadata: {} + name: wind-test + uid: 24bb3c33-6e93-4191-8dc6-24b3df7cb1ed + - conditions: + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: reconciled + reason: Succeeded + status: "True" + type: Ready + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: not cordoned + reason: Active + status: "False" + type: Cordoned + metadata: {} + name: wind-prod + uid: b3f3201b-8527-47c4-928b-ad6ae610e707 + state: Active +namespace: + apiVersion: v1 + kind: Namespace + metadata: + creationTimestamp: "2026-02-06T09:54:30Z" + labels: + capsule.clastix.io/tenant: wind + kubernetes.io/metadata.name: wind-test + name: wind-test + ownerReferences: + - apiVersion: capsule.clastix.io/v1beta2 + kind: Tenant + name: wind + uid: 93992a2b-cba4-4d33-9d09-da8fc0bfe93c + resourceVersion: "3977" + uid: 24bb3c33-6e93-4191-8dc6-24b3df7cb1ed + spec: + finalizers: + - kubernetes + status: + phase: Active +``` diff --git a/content/en/docs/replications/tenant.md b/content/en/docs/replications/tenant.md new file mode 100644 index 0000000..efc4e90 --- /dev/null +++ b/content/en/docs/replications/tenant.md @@ -0,0 +1,1247 @@ +--- +title: TenantResources +weight: 2 +description: > + Replicate resources across a Tenant's Namespaces as Tenant Owner +--- + +## Overview + +`TenantResource` is a namespace-scoped CRD that lets Tenant owners automatically replicate Kubernetes resources across all Namespaces in their Tenant - without manual distribution or custom automation. It is the tenant-level counterpart to [GlobalTenantResource](../global/), which is reserved for cluster administrators. + +The diagram below shows that an Administrator or a Tenant Owner can create a `TenantResource` inside a `Tenant`. In the `TenantResource` spec, a user specifies which resource they would like to replicate across the `Tenant`. When applied, this resource gets automatically distributed across all Namespaces that are part of the `Tenant`. + +![Tenant Resource Replication overview](/images/content/replication-tenantresource.png) + +## Prerequisites + +Tenant owners must have RBAC permission to create, update, and delete `TenantResource` objects. The following `ClusterRole` aggregates to the `admin` role, granting all holders permission to manage `TenantResource` instances: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: allow-tenant-resources + labels: + rbac.authorization.k8s.io/aggregate-to-admin: "true" +rules: +- apiGroups: ["capsule.clastix.io"] + resources: ["tenantresources"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] +``` + +## Example + +Alice, the project lead for the `solar` tenant, wants to provision a PostgreSQL database for each production Namespace automatically: + +```bash +$ kubectl get namespaces -l capsule.clastix.io/tenant=solar --show-labels +NAME STATUS AGE LABELS +solar-1 Active 59s capsule.clastix.io/tenant=solar,environment=production,kubernetes.io/metadata.name=solar-1,name=solar-1 +solar-2 Active 58s capsule.clastix.io/tenant=solar,environment=production,kubernetes.io/metadata.name=solar-2,name=solar-2 +solar-system Active 62s capsule.clastix.io/tenant=solar,kubernetes.io/metadata.name=solar-system,name=solar-system +``` + +She creates a `TenantResource` in `solar-system`: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: solar-db + namespace: solar-system +spec: + resyncPeriod: 60s + resources: + - additionalMetadata: + labels: + "replicated-by": "capsule" + namespaceSelector: + matchLabels: + environment: production + rawItems: + - apiVersion: postgresql.cnpg.io/v1 + kind: Cluster + metadata: + name: "postgres-{{namespace}}" + spec: + description: PostgreSQL cluster for the {{tenant.name}} Project + instances: 3 + postgresql: + pg_hba: + - hostssl app all all cert + primaryUpdateStrategy: unsupervised + storage: + size: 1Gi +``` + +Capsule replicates the `Cluster` resource into every Namespace matching the `namespaceSelector`. The Namespace where the `TenantResource` itself lives (`solar-system`) is automatically excluded, and Capsule injects labels to prevent the `TenantResource` from propagating into unowned Namespaces. + +```bash +$ kubectl get clusters.postgresql.cnpg.io -A +NAMESPACE NAME AGE INSTANCES READY STATUS PRIMARY +solar-1 postgres-solar-1 80s 3 3 Cluster in healthy state postgresql-1 +solar-2 postgres-solar-2 80s 3 3 Cluster in healthy state postgresql-1 +``` + +> Objects managed by this controller can be either **created** (new objects) or **adopted** (existing objects). See [Object Management](#object-management) in the Advanced section for full details. + +--- + +## Basic Usage + +### Resources + +A resource block defines *what* to replicate. Multiple blocks can be stacked in the `resources` array, each using one or more of the strategies below. + +#### NamespaceSelector + +The `namespaceSelector` field restricts replication to Namespaces matching a label selector. Capsule also protects selected resources from modification by Tenant users via its webhook. + +#### AdditionalMetadata + +Use `additionalMetadata` to attach extra `labels` and `annotations` to every generated object. [Fast Template values](/docs/operating/templating/#fast-templates) are supported: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-cluster-rbac +spec: + scope: Tenant + resources: + - additionalMetadata: + labels: + k8s.company.com/tenant: "{{tenant.name}}" + annotations: + k8s.company.com/cost-center: "inv-120" + generators: + - missingKey: error + template: | + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: tenant:{{$.tenant.metadata.name}}:priority + labels: + k8s.company.com/tenant: "test" + rules: + - apiGroups: ["scheduling.k8s.io"] + verbs: ["get"] + resources: ["priorityclasses"] +``` + +When the same label key appears in both `additionalMetadata` and the template, `additionalMetadata` takes priority. + +The following labels are always stripped because they are reserved for the controller: + + * `capsule.clastix.io/resources` + * `projectcapsule.dev/created-by` + * `capsule.clastix.io/managed-by` + * `projectcapsule.dev/managed-by` + +#### NamespacedItems + +Reference existing resources for replication across Tenant Namespaces. The controller validates that any resource kind listed here is namespace-scoped; cluster-scoped kinds are rejected with an error. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + + # Replicate all Configmaps labeled with projectcapsule.dev/replicate: "true" + - apiVersion: v1 + kind: ConfigMap + selector: + matchLabels: + projectcapsule.dev/replicate: "true" + + # Replicate all Configmaps labeled with projectcapsule.dev/replicate: "true" and in namespace capsule-system + - apiVersion: v1 + kind: ConfigMap + namespace: capsule-system + selector: + matchLabels: + projectcapsule.dev/replicate: "true" + + # Replicate Configmap named "logging-config" in namespace capsule-system labeled with projectcapsule.dev/replicate: "true" and in namespace capsule-system + - apiVersion: v1 + kind: ConfigMap + name: logging-config + namespace: capsule-system +``` + +**Note**: Resources with the label `projectcapsule.dev/created-by: resources` are ignored by `namespacedItems` to prevent reconciliation loops. + +If you try to define a cluster-scoped resource under `namespacedItems`, the reconciliation will fail immediately: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + - apiVersion: addons.projectcapsule.dev/v1alpha1 + kind: SopsProvider + name: infrastructure-provider + optional: true + +status: + conditions: + - lastTransitionTime: "2026-01-15T21:04:15Z" + message: cluster-scoped kind addons.projectcapsule.dev/v1alpha1/SopsProvider is + not allowed + reason: Failed + status: "False" + type: Ready +``` + +##### Name + +Providing `name` triggers a `GET` request for that single resource rather than a `LIST`: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-resource-replications + namespace: wind-test +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + # Fetch ConfigMaps labeled with the tenant name and replicate them into each Tenant Namespace + - apiVersion: v1 + kind: ConfigMap + name: "logging-config" +``` + +This distributes the `ConfigMap` named `logging-config` to all other Namespaces of the Tenant that `wind-test` belongs to. + +[Fast Templates](/docs/operating/templating/#fast-templates) are supported for `name`, `namespace`, and `selector`. + +##### Namespace + +Providing only `namespace` performs a `LIST` of all resources of that kind in that namespace: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-resource-replications +spec: + resources: + - namespacedItems: + - apiVersion: v1 + kind: ConfigMap + name: config-namespace + optional: true +``` + +[Fast Templates](/docs/operating/templating/#fast-templates) are supported for the `namespace` property: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + # Fetch ConfigMaps labeled with the tenant name and replicate them into each Tenant Namespace + - apiVersion: v1 + kind: Secret + namespace: "{{tenant.name}}-system" +``` + +##### Selector + +When using `selector`, the selector labels are stripped from the replicated objects. This prevents the replicated copy from also matching the source selector, which would cause a circular reconciliation loop. + +Source `ConfigMap`: + +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: app-config + labels: + projectcapsule.dev/replicate: "true" + namespace: wind-test +data: + player_initial_lives: "3" + ui_properties_file_name: "user-interface.properties" +``` + +`TenantResource`: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: app-config +spec: + resources: + - namespacedItems: + - apiVersion: v1 + kind: ConfigMap + selector: + matchLabels: + projectcapsule.dev/replicate: "true" +``` + +Resulting object in `wind-prod` (notice the absence of `projectcapsule.dev/replicate`): + +```yaml +apiVersion: v1 +data: + player_initial_lives: "3" + ui_properties_file_name: "user-interface.properties" +kind: ConfigMap +metadata: + labels: + projectcapsule.dev/created-by: resources + projectcapsule.dev/managed-by: resources + name: app-config + namespace: wind-prod + resourceVersion: "784529" + uid: 5f10a3f3-863e-4f45-9454-cff8f5bce86a +``` + +[Fast Templates](/docs/operating/templating/#fast-templates) are supported for `selector`: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 60s + resources: + - namespacedItems: + # Fetch ConfigMaps labeled with the tenant name and replicate them into each Tenant Namespace + - apiVersion: v1 + kind: ConfigMap + selector: + matchLabels: + company.com/replicate-for: "{{tenant.name}}" +``` + +#### Raw + +Raw items let you define resources inline as standard Kubernetes manifests. Use this when the resource does not yet exist in the cluster, or when you want to define it directly in the spec. [Fast Templates](/docs/operating/templating/#fast-templates) are supported. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-resource-replications +spec: + resyncPeriod: 300s + resources: + - rawItems: + - apiVersion: v1 + kind: LimitRange + metadata: + name: "{{tenant.name}}-{{namespace}}-resource-constraint" + spec: + limits: + - default: # this section defines default limits + cpu: 500m + defaultRequest: # this section defines default requests + cpu: 500m + max: # max and min define the limit range + cpu: "1" + min: + cpu: 100m + type: Container +``` + +The following example creates a [`SopsProvider`](https://github.com/peak-scale/sops-operator) for each Tenant: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-sops-providers +spec: + resyncPeriod: 600s + scope: Tenant + resources: + - rawItems: + - apiVersion: addons.projectcapsule.dev/v1alpha1 + kind: SopsProvider + metadata: + name: "{{tenant.name}}-secrets" + spec: + keys: + - namespaceSelector: + matchLabels: + capsule.clastix.io/tenant: "{{tenant.name}}" + sops: + - namespaceSelector: + matchLabels: + capsule.clastix.io/tenant: "{{tenant.name}}" +``` + +Because [Server-Side Apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/) is used, you only need to specify the fields you want to manage - the full resource spec is not required. + +For more advanced templating, consider [Generators](#generators). + +#### Generators + +Generators render one or more Kubernetes objects from a Go template string. The template content must be valid YAML; multi-document output separated by `---` is supported. The template engine is based on [go-sprout](https://github.com/go-sprout/sprout) - see [available functions](/docs/operating/templating/#sprout-templating). + +A simple example that creates a `ClusterRole` per Tenant: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-cluster-rbac +spec: + resources: + - generators: + - missingKey: error + template: | + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: tenant:{{$.tenant.metadata.name}}:reader + rules: + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "watch", "list"] +``` + +Templates can also produce multiple objects using flow control: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-priority-rbac +spec: + scope: Tenant + resources: + - generators: + - missingKey: error + template: | + {{- range $.tenant.status.classes.priority }} + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: tenant:{{$.tenant.metadata.name}}:priority:{{.}} + rules: + - apiGroups: ["scheduling.k8s.io"] + resources: ["priorityclasses"] + resourceNames: ["{{.}}"] + verbs: ["get"] + {{- end }} +``` + +See [Base Context](#base-context) for available template variables. To load additional resources into the template context, see [Context](#context) in the Advanced section. + +##### Template Snippets + +Some snippets that might be useful for certain cases. + +###### Names + +Extract the `Tenant` name: + +```html +{{ $.tenant.metadata.name }} +``` + +Extract the `Namespace` name: + +```html +{{ $.namespace.metadata.name }} +``` + +###### Foreach Owner + +Iterate over all owners of a Tenant: + +```html + {{- range $.tenant.status.owners }} + {{ .kind }}: {{ .name }} + {{- end }} +``` + +##### MissingKey + +Controls template behaviour when a referenced context key is absent. + +###### Invalid + +Continues execution silently. Missing keys render as the string `""`. + +This definition with the missing context: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: missing-key +spec: + resources: + - generators: + - missingKey: invalid + template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: show-key + data: + value: {{ $.custom.account.name }} +``` + +Turns into after templating: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: show-key +data: + value: "" +``` + +###### Zero + +**This is the default behavior.** Missing keys resolve to the zero value of their type (usually an empty string). + +This definition with the missing context: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: missing-key +spec: + resources: + - generators: + - missingKey: zero + template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: show-key + data: + value: {{ $.custom.account.name }} +``` + +Turns into after templating: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: show-key +data: + value: "" +``` + +###### Error + +Stops execution immediately with an error when a required key is missing. + +This definition with the missing context: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: missing-key +spec: + resources: + - generators: + - missingKey: error + template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: show-key + data: + value: {{ $.custom.account.name }} +``` + +Will error the `TenantResources`: + +```sh +NAME ITEMS READY STATUS AGE +missing-key 6 False error running generator: template: tpl:7:13: executing "tpl" at <$.custom.account.name>: map has no entry for key "custom" 9m5s +``` + +--- + +### Reconciliation + +#### Period + +`TenantResources` reconcile on the interval defined by `resyncPeriod`. The default is `60s`. Capsule does not watch source resources for changes; it reconciles periodically. A very short interval on large clusters with many Tenants and Namespaces can cause performance issues - tune this value accordingly. + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: renewable-pull-secrets + namespace: wind-test +spec: + resyncPeriod: 300s # 5 minutes + resources: + - namespacedItems: + - apiVersion: v1 + kind: Secret + namespace: harbor-system + selector: + matchLabels: + tenant: renewable +``` + +#### Manual + +To trigger an immediate reconciliation, add the `reconcile.projectcapsule.dev/requestedAt` annotation. The annotation is removed once reconciliation completes, making the process repeatable. + +```bash +kubectl annotate tenantresource renewable-pull-secrets -n wind-test \ + reconcile.projectcapsule.dev/requestedAt="$(date -Iseconds)" +``` + +--- + +### Impersonation + +{{% alert title="Information" color="warning" %}} +Without a configured ServiceAccount, the Capsule controller ServiceAccount is used for replication operations. This may allow privilege escalation if the controller has broader permissions than Tenant owners. +{{% /alert %}} + +Enabling impersonation ensures that replication operations run under a specific ServiceAccount identity, providing a proper audit trail and limiting privilege exposure. You can check which ServiceAccount is currently in use via the object's status: + +```bash +kubectl get tenantresource custom-cm -o jsonpath='{.status.serviceAccount}' | jq +{ + "name": "capsule", + "namespace": "capsule-system" +} +``` + +To use a different ServiceAccount, set the `serviceAccount` field on the object. For `TenantResource`, only the name is required - the namespace is always inferred from the Namespace the `TenantResource` resides in: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-resource-replications + namespace: wind-test +spec: + serviceAccount: + name: "default" + resources: + - namespacedItems: + - apiVersion: v1 + kind: ConfigMap + name: "config-namespace" +``` + +If the ServiceAccount lacks the required RBAC, replication will fail with a permission error: + +``` + - kind: ConfigMap + name: game-demo + namespace: wind-test + status: + created: true + message: 'apply failed for item 0/raw-0: applying object failed: configmaps + "game-demo" is forbidden: User "system:serviceaccount:wind-test:default" + cannot patch resource "configmaps" in API group "" in the namespace "wind-test"' + status: "False" + type: Ready + tenant: wind + version: v1 +``` + +Grant the ServiceAccount the necessary permissions: + +```yaml +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: capsule-tenant-replications +rules: +- apiGroups: [""] + resources: ["configmaps"] + verbs: ["list", "get", "patch", "create", "delete"] +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: default-sa-replication +spec: + resyncPeriod: 60s + resources: + - rawItems: + - apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: wind-replication + subjects: + - kind: ServiceAccount + name: default + namespace: wind-test + roleRef: + kind: ClusterRole + name: capsule-tenant-replications + apiGroup: rbac.authorization.k8s.io +``` + +#### Required Permissions + +The following permissions are required for each resource type managed by the replication feature: + + * `get` (always required) + * `create` (always required) + * `patch` (always required) + * `delete` (always required) + * `list` (required for [Namespaced Items](#namespaceditems) and [Context](#context)) + +Missing any of these will cause replication to fail. + +#### Default ServiceAccount + +To ensure all `TenantResource` objects use a controlled identity by default, configure a default ServiceAccount in the Capsule manager options. Per-object `serviceAccount` fields override this default. Only the name is required; the namespace is always the one the `TenantResource` resides in. + +[Read more about Impersonation](/docs/operating/setup/configuration/#impersonation). + +```yaml +manager: + options: + impersonation: + tenantDefaultServiceAccount: "default" +``` + +The default ServiceAccount must have sufficient RBAC. You can use a [GlobalTenantResource](../global/) to distribute the required `RoleBinding` across all Tenants: + +```yaml +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: capsule-default-namespace +rules: +- apiGroups: [""] + resources: ["limitranges", "secrets"] + verbs: ["get", "patch", "create", "delete", "list"] +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: default-sa-replication +spec: + resyncPeriod: 60s + resources: + - rawItems: + - apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: default-replication + subjects: + - kind: ServiceAccount + name: default + namespace: wind-test + roleRef: + kind: ClusterRole + name: capsule-tenant-replications + apiGroup: rbac.authorization.k8s.io +``` + +--- + +## Advanced + +This section covers more advanced features of the Replication setup. + +### Object Management + +Capsule uses [Server-Side Apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/) for all replication operations. Two management modes exist depending on whether the object already existed before reconciliation. + +#### Create + +An object is *Created* when the `TenantResource` first encounters it - it did not exist prior to reconciliation. Created objects receive the following metadata: + + * `metadata.labels.projectcapsule.dev/created-by`: `resources` + * `metadata.labels.projectcapsule.dev/managed-by`: `resources` + +```yaml +kind: ConfigMap +metadata: + labels: + projectcapsule.dev/created-by: resources + projectcapsule.dev/managed-by: resources + name: common-config + namespace: wind-test + resourceVersion: "549517" + uid: 23abbb7a-2926-416a-bc72-9f793ebf6080 +``` + +Because Server-Side Apply tracks field ownership, multiple `TenantResource` objects can contribute non-conflicting fields to the same object: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-ns-cm-registration + namespace: wind-test +spec: + resources: + - generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: common-config + data: + {{ $.namespace.metadata.name }}.conf: | + {{ toYAML $.namespace .metadata | nindent 4 }} + - rawItems: + - apiVersion: v1 + kind: ConfigMap + metadata: + name: common-config + data: + additional-data: "raw" +``` + +Result: + +```yaml +apiVersion: v1 +data: + additional-data: raw + wind-test.conf: |2 + creationTimestamp: "2026-02-10T10:58:33Z" + labels: + capsule.clastix.io/tenant: wind + kubernetes.io/metadata.name: wind-test + name: wind-test + ownerReferences: + - apiVersion: capsule.clastix.io/v1beta2 + kind: Tenant + name: wind + uid: 42f72944-f6d9-44a2-9feb-cd2b52f4043d + resourceVersion: "526252" + uid: 3f280d61-98b7-4188-9853-9a6598ca10a9 +kind: ConfigMap +metadata: + creationTimestamp: "2026-02-05T15:37:09Z" + labels: + projectcapsule.dev/created-by: resources + projectcapsule.dev/managed-by: resources + name: common-config + namespace: wind-test + resourceVersion: "561707" + uid: 33cfe1c6-1c9e-4417-9dd5-26ac0ba3bc85 +``` + +You can check the `created` property on each item's status to determine whether it was created or adopted. Field conflicts can be resolved with [Force](#force). + +##### Pruning + +When pruning is enabled, *Created* objects are deleted when they fall out of scope. When pruning is disabled, the `metadata.labels.projectcapsule.dev/managed-by` label is removed instead. + +The label `metadata.labels.projectcapsule.dev/created-by` is preserved after pruning, allowing another `GlobalTenantResource` or `TenantResource` to take ownership without explicit adoption. To prevent re-adoption, remove or change this label manually. + +#### Adopt + +By default, a `TenantResource` cannot modify objects it did not create. Adoption must be explicitly enabled. Adopted objects receive the following metadata: + + * `metadata.labels.projectcapsule.dev/managed-by`: `resources` + +The following example attempts to modify the existing `app-demo` `ConfigMap` in `wind-test`: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: app-config + namespace: wind-test +spec: + resources: + - generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: app-demo + data: + {{ $.namespace.metadata.name }}.conf: | + {{ toYAML $.namespace .metadata | nindent 4 }} +``` + +Without adoption enabled, all items fail: + +```yaml +kubectl get tenantresource argo-cd-permission -o yaml + +... + processedItems: + - kind: ConfigMap + name: app-demo + namespace: wind-prod + origin: 0/template-0-0 + status: + created: true + lastApply: "2026-02-10T17:59:46Z" + status: "True" + type: Ready + tenant: wind + version: v1 + - kind: ConfigMap + name: app-demo + namespace: wind-test + origin: 0/template-0-0 + status: + message: 'apply failed for item 0/template-0-0: evaluating managed metadata: + object v1/ConfigMap wind-test/app-demo exists and cannot be adopted' + status: "False" + type: Ready + tenant: wind + version: v1 +``` + +Enable adoption by setting `settings.adopt: true`: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: app-config + namespace: wind-test +spec: + settings: + adopt: true + resources: + - generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: app-demo + data: + {{ $.namespace.metadata.name }}.conf: | + {{ toYAML $.namespace .metadata | nindent 4 }} +``` + +When adoption is enabled, resources can be modified. Note that if multiple operators manage the same resource, all must use Server-Side Apply to avoid conflicts. + +```yaml + processedItems: + - kind: ConfigMap + name: app-demo + namespace: wind-prod + origin: 0/generator-0-0 + status: + created: true + lastApply: "2026-02-10T17:59:46Z" + status: "True" + type: Ready + tenant: wind + version: v1 + - kind: ConfigMap + name: app-demo + namespace: wind-test + origin: 0/generator-0-0 + status: + lastApply: "2026-02-10T18:01:31Z" + status: "True" + type: Ready + tenant: wind + version: v1 +``` + +##### Pruning + +When pruning is enabled, adoption is reverted - the patches introduced by the `TenantResource` are removed from the object. When pruning is disabled, only the `metadata.labels.projectcapsule.dev/managed-by` label is removed. + +--- + +### DependsOn + +A `TenantResource` can declare dependencies on other `TenantResource` objects in the same Namespace using `dependsOn`. The controller will not reconcile the resource until all declared dependencies are in `Ready` state. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: gitops-secret + namespace: wind-test +spec: + resyncPeriod: 60s + dependsOn: + - name: custom-cm + resources: + - additionalMetadata: + labels: + projectcapsule.dev/tenant: "{{tenant.name}}" + rawItems: + - apiVersion: v1 + kind: Secret + metadata: + name: myregistrykey + namespace: awesomeapps + data: + .dockerconfigjson: UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg== + type: kubernetes.io/dockerconfigjson +``` + +The status reflects whether a dependency is not yet ready: + +```bash +kubectl get tenantresource -n wind-test + +NAME ITEM COUNT READY STATUS AGE +custom-cm 6 False applying of 6 resources failed 12h +gitops-secret 6 False dependency custom-cm not ready 8h +``` + +If a dependency does not exist: + +```bash +kubectl get tenantresource gitops-secret -n wind-test + +NAME ITEM COUNT READY STATUS AGE +gitops-secret 6 False dependency custom-cm not found 8h +``` + +Dependencies are evaluated in the order they are declared in the `dependsOn` array. + +--- + +### Force + +Setting `settings.force: true` instructs Capsule to [force-apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts) changes on Server-Side Apply conflicts, claiming field ownership even if another manager already holds it. + +**This option should generally be avoided.** Forcing ownership over a field managed by another operator will almost certainly cause a reconcile war. Only use it in scenarios where you intentionally want Capsule to win ownership disputes. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-technical-accounts + namespace: wind-test +spec: + settings: + force: true + resources: + - generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: shared-config + data: + common.conf: | + {{ toYAML $.tenant.metadata | nindent 4 }} +``` + +--- + +### Context + +The `context` field lets you load additional Kubernetes resources into the template rendering context. This is useful when you need to iterate over existing objects as part of your template logic: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: tenant-sops-providers +spec: + resyncPeriod: 600s + resources: + - context: + resources: + - index: secrets + apiVersion: v1 + kind: Secret + namespace: "{{.namespace}}" + selector: + matchLabels: + pullsecret.company.com: "true" + - index: sa + apiVersion: v1 + kind: ServiceAccount + namespace: "{{.namespace}}" + + generators: + - template: | + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: show-context + data: + context.yaml: | + {{- toYAML $ | nindent 4 }} +``` + +#### Base Context + +The following context is always available in generator templates. The `tenant` key is always present. + +```yaml +tenant: + apiVersion: capsule.clastix.io/v1beta2 + kind: Tenant + metadata: + creationTimestamp: "2026-02-06T09:54:30Z" + generation: 1 + labels: + kubernetes.io/metadata.name: wind + name: wind + resourceVersion: "4038" + uid: 93992a2b-cba4-4d33-9d09-da8fc0bfe93c + spec: + additionalRoleBindings: + - clusterRoleName: view + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: wind-users + owners: + - clusterRoles: + - admin + - capsule-namespace-deleter + kind: User + name: joe + permissions: + matchOwners: + - matchLabels: + team: devops + - matchLabels: + tenant: wind + status: + classes: + priority: + - system-cluster-critical + - system-node-critical + storage: + - standard + conditions: + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: reconciled + reason: Succeeded + status: "True" + type: Ready + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: not cordoned + reason: Active + status: "False" + type: Cordoned + namespaces: + - wind-prod + - wind-test + owners: + - clusterRoles: + - admin + - capsule-namespace-deleter + kind: Group + name: oidc:org:devops + - clusterRoles: + - admin + - capsule-namespace-deleter + kind: User + name: joe + size: 2 + spaces: + - conditions: + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: reconciled + reason: Succeeded + status: "True" + type: Ready + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: not cordoned + reason: Active + status: "False" + type: Cordoned + metadata: {} + name: wind-test + uid: 24bb3c33-6e93-4191-8dc6-24b3df7cb1ed + - conditions: + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: reconciled + reason: Succeeded + status: "True" + type: Ready + - lastTransitionTime: "2026-02-06T09:54:30Z" + message: not cordoned + reason: Active + status: "False" + type: Cordoned + metadata: {} + name: wind-prod + uid: b3f3201b-8527-47c4-928b-ad6ae610e707 + state: Active + + +namespace: + apiVersion: v1 + kind: Namespace + metadata: + creationTimestamp: "2026-02-06T09:54:30Z" + labels: + capsule.clastix.io/tenant: wind + kubernetes.io/metadata.name: wind-test + name: wind-test + ownerReferences: + - apiVersion: capsule.clastix.io/v1beta2 + kind: Tenant + name: wind + uid: 93992a2b-cba4-4d33-9d09-da8fc0bfe93c + resourceVersion: "3977" + uid: 24bb3c33-6e93-4191-8dc6-24b3df7cb1ed + spec: + finalizers: + - kubernetes + status: + phase: Active +``` + diff --git a/content/en/docs/resourcepools/_index.md b/content/en/docs/resourcepools/_index.md index c71d702..d345fdd 100644 --- a/content/en/docs/resourcepools/_index.md +++ b/content/en/docs/resourcepools/_index.md @@ -332,17 +332,85 @@ If no Pool can be auto-assigned, the `ResourcePoolClaim` will enter an `Unassign The Auto-Assignment Process is only executed, when `.spec.pool` is unset on `Create` or `Update` operations. -#### Release +#### Bound -If a `ResourcePoolClaim` is deleted, the resources are released back to the `ResourcePool`. This means that the resources are no longer reserved for the claim and can be used by other claims. Releasing can be achieved : +A `ResourcePoolClaim` is considered `Bound`, when the requested resources from the claim were successfully allocated from the `ResourcePool`. And the resources are actually used by any `ResourceQuota` in the namespace the claim was created in. If the resources are not used yet, the `ResourcePoolClaims` is considered `Unused` and can be deleted, change to a different `ResourcePool` or released without any further actions. However when it's resources are used, the claim is `Bound` and can not be modified or deleted until the resources are released (not longer in use). -- By deleting the `ResourcePoolClaim` object. -- By annotating the `ResourcePoolClaim` with `projectcapsule.dev/release: "true"`. This will release the `ResourcePoolClaim` from the `ResourcePool` without deleting the object itself and instantly requeue. +The selection of which `ResourcePoolClaim` is `Bound` is based on a greedy pattern. Meaning we sort the `ResourcePoolClaims` by their `CreationTimestamp` and try to allocate them one by one until no more resources are available from the `ResourcePool`. + +Let's see this in action. We can see that both claims are unused and can be released. + + +```shell +kubectl get resourcepoolclaim -n solar-test + +NAME POOL READY MESSAGE BOUND REASON AGE +get-me-solar solar-pool True reconciled False claim is unused 9h +get-me-solar-2 solar-pool True reconciled False claim is unused 9h + + +kubectl get resourcequota -n solar-test + +NAME REQUEST LIMIT AGE +capsule-pool-solar-pool requests.cpu: 4/4, requests.memory: 4Gi/4Gi 7m53s + +``` + +We now create a pod to consume the amount of resources provided by the claim `get-me-solar` (`cpu: 2` and `memory: 2Gi`). We can see that half of the claim is now used: + +```shell +kubectl get resourcepoolclaim -n solar-test + +NAME POOL READY MESSAGE BOUND REASON AGE +get-me-solar solar-pool True reconciled True claim is used 12m +get-me-solar-2 solar-pool True reconciled False claim is unused 12m + +kubectl get resourcequota -n solar-test + +NAME REQUEST LIMIT AGE +capsule-pool-solar-pool requests.cpu: 2/4, requests.memory: 2Gi/4Gi 11m +``` + +We can remove `get-me-solar-2`, as it's still unused: ```shell -kubectl annotate resourcepoolclaim skip-the-line -n solar-prod projectcapsule.dev/release="true" +kubectl delete resourcepoolclaim -n solar-test get-me-solar-2 + +resourcepoolclaim.capsule.clastix.io "get-me-solar-2" deleted +``` + +However interactions with `get-me-solar` are now limited, as it's `Bound`: + +```shell +kubectl delete resourcepoolclaim -n solar-test get-me-solar + +Error from server (Forbidden): admission webhook "resourcepoolclaims.projectcapsule.dev" denied the request: cannot delete the pool while claim is used in resourcepool solar-pool +``` + +If we remove the pod again, the `ResourcePoolClaim` becomes unused again and can be deleted or modified. + +```shell +kubectl get resourcepoolclaim -n solar-test + +NAME POOL READY MESSAGE BOUND REASON AGE +get-me-solar solar-pool True reconciled False claim is unused 16m + + +kubectl get resourcequota -n solar-test + +NAME REQUEST LIMIT AGE +capsule-pool-solar-pool requests.cpu: 0/2, requests.memory: 0/2Gi 17m ``` +#### Release + +If a `ResourcePoolClaim` is deleted, the resources are released back to the `ResourcePool`. This means that the resources are no longer reserved for the claim and can be used by other claims. + +- By deleting the `ResourcePoolClaim` object (**Recommended**). +- By annotating the `ResourcePoolClaim` with `projectcapsule.dev/release: "true"`. This will release the `ResourcePoolClaim` from the `ResourcePool` without deleting the object itself and instantly requeue. + +Both these actions can only be performed if the `ResourcePoolClaim` is in a [`Bound`](#bound) state `False` (not used currently). Otherwise your first have to free the resources used by the claim in order to release it. You can verify the `Bound` state for all `ResourcePoolClaims` in a namespace with. + #### Immutable Once a `ResourcePoolClaim` has successfully claimed resources from a `ResourcePool`, the claim is immutable. This means that the claim cannot be modified or deleted until the resources have been released back to the `ResourcePool`. This means `ResourcePoolClaim` can not be expanded or shrunk, without [releasing](#release). diff --git a/content/en/docs/tenants/administration.md b/content/en/docs/tenants/administration.md index b53551c..1761166 100644 --- a/content/en/docs/tenants/administration.md +++ b/content/en/docs/tenants/administration.md @@ -14,7 +14,23 @@ Bill needs to cordon a `Tenant` and its `Namespaces` for several reasons: * During incidents or outages * During planned maintenance of a dedicated nodes pool in a BYOD scenario -With this said, the `TenantOwner` and the related Service Account living into managed `Namespaces`, cannot proceed to any update, create or delete action. +With the default installation of Capsule all `CREATE`, `UPDATE` and `DELETE` operations performed by **[Capsule Users](/docs/operating/architecture/#capsule-users)** are droped. Any Updates to Subresources (i.e. `status` updates) and events are allowed to proceed as usual. If you wish to allow specific Operations, you can change the values for the Cordoning Admission via Values (eg. allow `Pod/DELETE` operations): + +```yaml +webhooks: + hooks: + cordoning: + matchConditions: + + - name: skip-pod-create-delete + expression: '!(request.resource.resource == "pods" && request.operation in ["DELETE"])' + + # Default conditions to ignore subresources and events + - name: ignore-subresources + expression: '!has(request.subResource) || request.subResource == ""' + - name: ignore-events + expression: 'request.resource.resource != "events"' +``` This is possible by just toggling the specific `Tenant` specification: @@ -30,7 +46,13 @@ spec: name: alice ``` -Any operation performed by Alice, the `TenantOwner`, will be rejected by the Admission controller. +Any operation performed by Alice, the `TenantOwner`, will be rejected by the Admission controller: + +```bash +kubectl delete pod --all -n solar-test --as alice --as-group projectcapsule.dev + +Error from server (Forbidden): admission webhook "cordoning.misc.projectcapsule.dev" denied the request: The current namespace 'solar-test' is cordoned. The attempted operation DELETE for /v1/Pod/nginx-deployment-56f567c7cb-pj86t is not permitted during cordoning status. +``` Uncordoning can be done by removing the said specification key: diff --git a/content/en/docs/tenants/enforcement.md b/content/en/docs/tenants/enforcement.md index a3e211f..2c92513 100644 --- a/content/en/docs/tenants/enforcement.md +++ b/content/en/docs/tenants/enforcement.md @@ -1,6 +1,6 @@ --- title: Enforcement -weight: 5 +weight: 6 description: > Configure policies and restrictions on tenant-basis --- diff --git a/content/en/docs/tenants/metadata.md b/content/en/docs/tenants/metadata.md index 3f66cbe..281db0f 100644 --- a/content/en/docs/tenants/metadata.md +++ b/content/en/docs/tenants/metadata.md @@ -1,20 +1,73 @@ --- title: Metadata -weight: 5 +weight: 7 description: > Inherit additional metadata on Tenant resources. --- ## Managed -By default all namespaced resources within a `Namespace` which are part of a `Tenant` labeled at admission with the `capsule.clastix.io/tenant: ` label. +By default all namespaced resources within a `Namespace` which are part of a `Tenant` labeled at admission with the following labels: + + * `capsule.clastix.io/managed-by`: `` (Legacy label) + * `projectcapsule.dev/tenant`: `` + +The labels are used by Capsule to identify resources belonging to a specific tenant. This is currently important for the [Capsule Proxy](/docs/proxy/) to filter resources accordingly. ## Namespaces + +### RequiredMetadata + +The cluster admin can enforce tenant owners to add specific metadata as `Labels` and `Annotations` to the `Namespaces` they create. This is a useful feature to enforce a set of [Rules](/docs/tenants/rules/) based on `Labels`. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + namespaceOptions: + requiredMetadata: + labels: + env: "^(prod|test|dev)$" + annotations: + example.corp/cost-center: "^INV-[0-9]{4}$" +``` + +If you add these properties to a `Tenant`, and there's already a `Namespace` in that `Tenant` that does not comply with the required metadata, the `Namespace` will have admission errors until the required metadata is added to it. + +Example with [Rules](/docs/tenants/rules/): + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + namespaceOptions: + requiredMetadata: + labels: + env: "^(prod|test|dev)$" + annotations: + example.corp/cost-center: "^INV-[0-9]{4}$" + + rules: + # Select a subset of namespaces (enviornment=prod) to allow further registries + - namespaceSelector: + matchExpressions: + - key: env + operator: In + values: ["prod"] + enforce: + registries: + - url: "harbor/v2/prod-registry/.*" + policy: [ "ifNotPresent" ] +``` + ### AdditionalMetadataList -{{% alert title="Information" color="info" %}} -Starting from v0.10.8, it is possible to use templated values for labels and annotations. -Currently, `{{ tenant.name }}` and `{{ namespace }}` placeholders are available. -{{% /alert %}} + ```yaml apiVersion: capsule.clastix.io/v1beta2 kind: Tenant @@ -31,6 +84,7 @@ spec: labels: templated-label: {{ namespace }} ``` + The cluster admin can "taint" the namespaces created by tenant owners with additional metadata as labels and annotations. There is no specific semantic assigned to these labels and annotations: they will be assigned to the namespaces in the tenant as they are created. However you have the option to be more specific by selecting to which namespaces you want to assign what kind of metadata: ```yaml diff --git a/content/en/docs/tenants/permissions.md b/content/en/docs/tenants/permissions.md index 241c7ec..07d5dff 100644 --- a/content/en/docs/tenants/permissions.md +++ b/content/en/docs/tenants/permissions.md @@ -5,6 +5,11 @@ description: > Grant permissions for tenants --- +## Administrators + +Administrators are users that have full control over all `Tenants` and their namespaces. They are typically cluster administrators or operators who need to manage the entire cluster and all its `Tenants`. However as administrator you are automatically Owner of all `Tenants`.`Tenants` This means that administrators can create, delete, and manage namespaces and other resources within any `Tenant`, given you are using [label assignments for tenants](/docs/tenants/namespaces/#label). + + ## Ownership Capsule introduces the principal, that tenants must have owners ([Tenant Owners](/docs/operating/architecture/#tenant-owners)). The owner of a tenant is a user or a group of users that have the right to create, delete, and manage the [tenant's namespaces](/docs/tenants/namespaces) and other tenant resources. However an owner does not have the permissions to manage the tenants they are owner of. This is still done by cluster-administrators. @@ -46,7 +51,6 @@ To explain these entries, let's inspect one of them: With this information available you - ### Tenant Owners Tenant Owners can be declared as dedicated cluster scoped Resources called `TenantOwner`. This allows the cluster admin to manage the ownership of tenants in a more flexible way, for example by adding labels and annotations to the `TenantOwner` resources. @@ -105,13 +109,10 @@ status: This can also be combined with direct owner declarations. In the example, both `alice` user and all `TenantOwners` with label `team: devops` and `TenantOwners` with label `customer: x` will be owners of the `solar` tenant. -```yaml ```yaml apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: - labels: - kubernetes.io/metadata.name: oil name: solar spec: owners: @@ -181,6 +182,71 @@ status: We. can see that the `system:serviceaccount:capsule:controller` ServiceAccount now has additional `mega-admin` and `controller` roles assigned. +#### Implicit Tenant Assignment + +If a `TenantOwner` is created all `Tenants` are always matching the label `projectcapsule.dev/tenant` on `TenantOwner` with the name of the `Tenant`. This means that if you create a `TenantOwner` with the name solar, it will automatically become owner of the solar `Tenant`. This can only be done for one tenant at a time (because the label is unique) and is intended that way. + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantOwner +metadata: + labels: + projectcapsule.dev/tenant: "solar" + name: solar-test-gitops-reconciler +spec: + kind: ServiceAccount + name: "system:serviceaccount:solar-test:gitops-reconciler" +``` + +With this `Tenant` specification: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: {} +``` + +We can observe that the owner is automatically assigned for the `Tenant` solar: + +```yaml +kubectl get tnt solar -o yaml + +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + cordoned: false + preventDeletion: false +status: + owners: + - clusterRoles: + - admin + - capsule-namespace-deleter + kind: ServiceAccount + name: system:serviceaccount:solar-test:gitops-reconciler +``` + +#### Aggregation + +All subjects defined in `TenantOwner` resources are automatically considered [Capsule Users](/docs/operating/architecture/#capsule-users) and don't need to mentioned further in the CapsuleConfiguration [User Scope](/docs/operating/setup/configuration/#users). If you don't want this behavior, you can disable it by setting `aggregate: false` in the `TenantOwner` spec: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantOwner +metadata: + labels: + customer: x + name: controller +spec: + kind: ServiceAccount + name: "system:serviceaccount:capsule:controller" + aggregate: false +``` + ### Users **Bill**, the cluster admin, receives a new request from Acme Corp's CTO asking for a new `Tenant` to be onboarded and Alice user will be the `TenantOwner`. Bill then assigns Alice's identity of alice in the Acme Corp. identity management system. Since Alice is a `TenantOwner`, Bill needs to assign alice the Capsule group defined by --capsule-user-group option, which defaults to `projectcapsule.dev`. @@ -239,8 +305,7 @@ no including the `Tenant` resources - -``` +```shell kubectl auth can-i get tenants no ``` @@ -314,28 +379,6 @@ system:serviceaccounts:{service-account-namespace} You have to add `system:serviceaccounts:{service-account-namespace}` to the CapsuleConfiguration [Group Scope](/docs/operating/setup/configuration/#usergroups) or `system:serviceaccounts:{service-account-namespace}:{service-account-name}` to the CapsuleConfiguration [User Scope](/docs/operating/setup/configuration/#usergroups) to make it work. -### ServiceAccount Promotion - -Within a `Tenant`, a ServiceAccount can be promoted to a `TenantOwner`. For example, Alice can create a ServiceAccount called robot in the solar `Tenant` and promote it to be a `TenantOwner` (This requires Alice to be an owner of the `Tenant` as well): - -```yaml -kubectl label sa gitops-reconcile -n green-test owner.projectcapsule.dev/promote=true --as alice --as-group projectcapsule.dev -``` - -Now the ServiceAccount robot can create namespaces in the solar `Tenant`: - -```bash -kubectl create ns green-valkey--as system:serviceaccount:green-test:gitops-reconcile -``` - -To revoke the promotion, Alice can just remove the label: - -```yaml -kubectl label sa gitops-reconcile -n green-test owner.projectcapsule.dev/promote- --as alice --as-group projectcapsule.dev -``` - -This feature must be enabled in the [CapsuleConfiguration](/docs/operating/setup/configuration/#allowserviceaccountpromotion). - ### Owner Roles By default, all `TenantOwners` will be granted with two ClusterRole resources using the RoleBinding API: @@ -579,6 +622,191 @@ spec: - tenant-resources ``` +## Promotion + +As [Tenant Owner](#ownership) you can perform `ServiceAccount` Promotion. + +### Owner Promotion + +Within a `Tenant`, a ServiceAccount can be promoted to a `TenantOwner`. For example, Alice can create a ServiceAccount called robot in the solar `Tenant` and promote it to be a `TenantOwner` (This requires Alice to be an owner of the `Tenant` as well): + +```yaml +kubectl label sa gitops-reconcile -n green-test owner.projectcapsule.dev/promote=true --as alice --as-group projectcapsule.dev +``` + +**Note:** Promotion is only triggered on the label `owner.projectcapsule.dev/promote` with the value `true` + +We can now verify if the promotion was successful by checking the `Tenant` status: + +```yaml +kubectl get tnt green -o jsonpath='{.status.owners}' | jq + +[ + { + "clusterRoles": [ + "capsule-namespace-provisioner", + "capsule-namespace-deleter" + ], + "kind": "ServiceAccount", + "name": "system:serviceaccount:green-test:gitops-reconcile" + }, + { + "clusterRoles": [ + "view", + "tenant-resources" + ], + "kind": "User", + "name": "joe" + } +] +``` + + +Now the ServiceAccount robot can create namespaces in the solar `Tenant`: + +```bash +kubectl create ns green-valkey--as system:serviceaccount:green-test:gitops-reconcile +``` + +To revoke the promotion, Alice can just remove the label: + +```yaml +kubectl label sa gitops-reconcile -n green-test owner.projectcapsule.dev/promote- --as alice --as-group projectcapsule.dev +``` + +This feature must be enabled in the [CapsuleConfiguration](/docs/operating/setup/configuration/#allowserviceaccountpromotion). The ClusterRoles assigned to promoted ServiceAccounts can be configured in the [CapsuleConfiguration](/docs/operating/setup/configuration/#rbac) as well. + +You can also dis/enable Owner Promotion per `Tenant`. By default it's enabled, however since it's disabled in the [CapsuleConfiguration](/docs/operating/setup/configuration/#allowserviceaccountpromotion) it can't be used, unless that's enabled as well. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + permissions: + promotions: + allowOwnerPromotion: false +``` + +### Rule Promotion + + +As an administrator, you can define promotion rules for each Tenant. A promotion rule selects ServiceAccounts within a Tenant based on specified conditions and assigns them predefined ClusterRoles. + +The selected ClusterRoles are then applied across all namespaces belonging to the Tenant, with the corresponding ServiceAccounts configured as subjects. This allows a ServiceAccount in one namespace to automatically receive equivalent permissions in all other namespaces of the same Tenant. + +This feature is particularly useful in scenarios involving [Tenant Replications](/docs/replications/#tenantresource), where consistent permissions across namespaces are required. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + permissions: + promotions: + rules: + + # With this rule every promoted ServiceAccount get's the ClusterRole "tenant-replicator" in all Namespaces + # of the Tenant solar + - clusterRoles: + - "configmap-replicator" + + # With this rule every promoted ServiceAccount with the matching labels get's the ClusterRole "tenant-replicator" in all Namespaces + # of the Tenant solar + - clusterRoles: + - "secret-replicator" + selector: + matchLabels: + super: "account" +``` + +Make sure the `ClusterRoles` exist, otherwise you will get a reconcile error for the corresponding `Tenant`: + +```shell + conditions: + - lastTransitionTime: "2026-02-16T23:08:59Z" + message: 'cannot sync rolebindings items: rolebindings.rbac.authorization.k8s.io + "tenant-replicator" not found' +``` + +If you are running capsule in [Strict Mode](/docs/operating/setup/installation/#strict-rbac) we must ensure the controller can grant the corresponding permissions to the `ServiceAccount` in all of the `Namespaces` in the `Tenant`. We can simply aggregate the same `ClusterRoles` to the controller: + +```yaml +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: configmap-replicator + labels: + projectcapsule.dev/aggregate-to-controller: "true" +rules: +- apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create", "patch", "watch", "list", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: secret-replicator + labels: + projectcapsule.dev/aggregate-to-controller: "true" +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "patch", "watch", "list", "delete"] +``` + +Now as [Tenant Owner](#ownership) we can start promoting `ServiceAccounts` by labeling them with the label `projectcapsule.dev/promote` and the value `true`. This feature must be enabled in the [CapsuleConfiguration](/docs/operating/setup/configuration/#allowserviceaccountpromotion). You will get the following admission error if the feature is disabled: + +```shell +Error from server (Forbidden): admission webhook "serviceaccounts.projectcapsule.dev" denied the request: service account promotion is disabled. Contact cluster administrators +``` + +When the feature is enabled the following command will succeded (assuming `alice` is a [Tenant Owner](#ownership) of the `Tenant` solar): + +```yaml +kubectl label sa gitops-reconcile -n solar-test projectcapsule.dev/promote=true --as alice --as-group projectcapsule.dev +``` + +We can now verify if the promotion was successful by checking the `Tenant` status: + +```yaml +kubectl get tnt solar -o jsonpath='{.status.promotions}' | jq + +[ + { + "clusterRoles": [ + "tenant-replicator" + ], + "kind": "ServiceAccount", + "name": "system:serviceaccount:solar-test:gitops-reconcile" + } +] +``` + +we can verify the rolebinding was distributed to other `Namespaces` of the `Tenant` solar: + +```shell +kubectl get rolebinding -n solar-prod + +NAME ROLE AGE +.. +capsule:managed:7ad688b586eada40 ClusterRole/configmap-replicator 21s +.. +``` + +To revoke the promotion, Alice can just remove the label: + +```yaml +kubectl label sa gitops-reconcile -n solar-test projectcapsule.dev/promote- --as alice --as-group projectcapsule.dev +``` + + + ## Additional Rolebindings With `Tenant` rolebindings you can distribute namespaced rolebindings to all namespaces which are assigned to a namespace. Essentially it is then ensured the defined rolebindings are present and reconciled in all namespaces of the `Tenant`. This is useful if users should have more insights on `Tenant` basis. Let's look at an example. @@ -625,6 +853,28 @@ EOF As you can see the subjects is a classic [rolebinding subject](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-subjects). This way you grant permissions to the subject user **Joe**, who only can list and watch servicemonitors in the solar tenant namespaces, but has no other permissions. +### Strict + +If you have [strict RBAC enabled for the controller](/docs/operating/setup/installation/#strict-rbac), you need to ensure that the controller ServiceAccount has the permission to create RoleBindings for the specified ClusterRole. The Controller Aggregates ClusterRoles with the labels (OR): + + - `projectcapsule.dev/aggregate-to-controller: "true"` + - `projectcapsule.dev/aggregate-to-controller-instance: {{ .Release.Name }}` + +So for the above example, you need to label the `prometheus-servicemonitors-viewer` ClusterRole like this: + +```yaml +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: prometheus-servicemonitors-viewer + labels: + projectcapsule.dev/aggregate-to-controller: "true" +rules: +- apiGroups: ["monitoring.coreos.com"] + resources: ["servicemonitors"] + verbs: ["get", "list", "watch"] +``` + ### Custom Resources Capsule grants admin permissions to the `TenantOwners` but is only limited to their namespaces. To achieve that, it assigns the ClusterRole [admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) to the `TenantOwner`. This ClusterRole does not permit the installation of custom resources in the namespaces. @@ -699,8 +949,3 @@ roleRef: With the above example, Capsule is leaving the `TenantOwner` to create namespaced custom resources. > Take Note: a `TenantOwner` having the admin scope on its namespaces only, does not have the permission to create Custom Resources Definitions (CRDs) because this requires a cluster admin permission level. Only Bill, the cluster admin, can create CRDs. This is a known limitation of any multi-tenancy environment based on a single shared control plane. - -## Administrators - -Administrators are users that have full control over all `Tenants` and their namespaces. They are typically cluster administrators or operators who need to manage the entire cluster and all its `Tenants`. However as administrator you are automatically Owner of all `Tenants`.`Tenants` This means that administrators can create, delete, and manage namespaces and other resources within any `Tenant`, given you are using [label assignments for tenants](/docs/tenants/namespaces/#label). - diff --git a/content/en/docs/tenants/quickstart.md b/content/en/docs/tenants/quickstart.md index b2dce61..06289aa 100644 --- a/content/en/docs/tenants/quickstart.md +++ b/content/en/docs/tenants/quickstart.md @@ -14,18 +14,66 @@ kind: Tenant metadata: name: oil spec: + permissions: + matchOwners: + - matchLabels: + team: platform owners: - name: alice - kind: User -EOF -``` + kind: Us You can check the tenant just created ```bash $ kubectl get tenants -NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE -solar Active 0 10s +NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STATUS AGE +oil Active 0 True reconciled 13s +``` + +We create dedicated `TenantOwners` who represent cluster administrators. They are matched by labels defined in the `permissions.matchOwners` section of the `Tenant` spec. In our case, any user or group with the label `team: platform` is considered a `TenantOwner` for the `oil` tenant. + +```bash +kubectl create -f - << EOF +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantOwner +metadata: + name: platform-team + labels: + team: platform + +spec: + kind: Group + name: "oidc:kubernetes:admin" +EOF +``` + +We can now verify all owners of the `oil` tenant: + +```bash +kubectl get tenant oil -o jsonpath='{.status.owners}' +``` + +The result should be similar to: + +```json +[ + { + "kind": "Group", + "name": "oidc:kubernetes:admin", + "clusterRoles": [ + "admin", + "capsule-namespace-deleter" + ] + }, + { + "kind": "User", + "name": "alice", + "clusterRoles": [ + "admin", + "capsule-namespace-deleter" + ] + } +] ``` ## Login as Tenant Owner @@ -40,10 +88,9 @@ For example, if you are using capsule.clastix.io, users authenticated through a Users authenticated through an OIDC token must have in their token: -``` -... +```json "users_groups": [ - "capsule.clastix.io", + "projectcapsule.dev", "other_group" ] ``` @@ -62,7 +109,6 @@ to use it as alice export KUBECONFIG=alice-solar.kubeconfig Login as tenant owner ```bash -$ export KUBECONFIG=alice-solar.kubeconfig ``` ### Impersonation @@ -70,7 +116,7 @@ $ export KUBECONFIG=alice-solar.kubeconfig You can simulate this behavior by using [impersonation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation): ```bash -kubectl --as alice --as-group capsule.clastix.io ... +kubectl --as alice --as-group projectcapsule.dev ... ``` ## Create namespaces @@ -78,22 +124,21 @@ kubectl --as alice --as-group capsule.clastix.io ... As tenant owner, you can create namespaces: ```bash -$ kubectl create namespace solar-production -$ kubectl create namespace solar-development +kubectl create namespace solar-production +kubectl create namespace solar-development ``` or - ```bash -$ kubectl --as alice --as-group capsule.clastix.io create namespace solar-production -$ kubectl --as alice --as-group capsule.clastix.io create namespace solar-development +kubectl --as alice --as-group projectcapsule.dev create namespace solar-production +kubectl --as alice --as-group projectcapsule.dev create namespace solar-development ``` And operate with fully admin permissions: ```bash -$ kubectl -n solar-development run nginx --image=docker.io/nginx -$ kubectl -n solar-development get pods +kubectl -n solar-development run nginx --image=docker.io/nginx +kubectl -n solar-development get pods ``` ## Limiting access diff --git a/content/en/docs/tenants/rules.md b/content/en/docs/tenants/rules.md new file mode 100644 index 0000000..cff1093 --- /dev/null +++ b/content/en/docs/tenants/rules.md @@ -0,0 +1,265 @@ +--- +title: Rules +weight: 5 +description: > + Configure policies and restrictions on tenant-basis with Rules +--- + +Enforcement rules allow Bill, the cluster admin, to set policies and restrictions on a per-`Tenant` basis. These rules are enforced by Capsule Admission Webhooks when Alice, the `TenantOwner`, creates or modifies resources in her `Namespaces`. With the Rule Construct we can profile namespaces within a tenant to adhere to specific policies, depending on metadata. + +## Namespace Selector + +By default a rule is applied to all namespaces within a `Tenant`. However you can select a subset of namespaces to apply the rule on, by using a `namespaceSelector`. This selector works the same way as a standard Kubernetes label selector: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + ... + rules: + # Matches all Namespaces and enforces the rule for all of them + - enforce: + registries: + - url: "harbor/v2/customer-registry/.*" + policy: [ "ifNotPresent" ] + + # Select a subset of namespaces (enviornment=prod) to allow further registries + - namespaceSelector: + matchExpressions: + - key: env + operator: In + values: ["prod"] + enforce: + registries: + - url: "harbor/v2/prod-registry/.*" + policy: [ "ifNotPresent" ] +``` + +Note that rules are combined together. In the above example, all namespaces within the `solar` tenant will be enforced to use images from `harbor/v2/customer-registry/*`, while namespaces labeled with `env=prod` will also be allowed to pull images from `harbor/v2/prod-registry/*`. + +## Enforcement + +Declare Enforcement rules for the selected namespaces. + +### Registries + +Define allowed image registries for `Pods` with rules. Each registry can have specific policies and validation targets. We use Regexp pattern matching for the registry URL, so you can specify patterns like `harbor/v2/customer-registry/*` to match all images from that registry. The matching is done against the full image name, including the path and tag. The ordering is based on the order of declaration, so the first matching rule will be applied. The later the match is found, the higher the precedence. This allows you to build constructs like these: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + ... + rules: + - enforce: + registries: + + # Enforce PullPolicy "always" for all registries (For Container Images and Volume Images) + - url: ".*" + policy: [ "Always" ] + + # If we are pulling from a harbor registry we want to verify the images for Containers only + - url: "harbor/.*" + + - namespaceSelector: + matchExpressions: + - key: env + operator: In + values: ["prod"] + enforce: + registries: + - url: "harbor/v2/customer-registry/prod-image/.*" + policy: [ "Always" ] +``` + +Let's try to apply the following pod in the namespace `solar-test` (Does not match the `env=prod` selector): + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: image-volume +spec: + containers: + + - name: shell + command: ["sleep", "infinity"] + imagePullPolicy: Never + image: harbor/v2/prod-registry/debian + volumeMounts: + - name: volume + mountPath: /volume + + volumes: + - name: volume + image: + reference: quay.io/crio/artifact:v2 + pullPolicy: IfNotPresent + + +``` + +**What do you expect to happen?** + +```bash +kubectl apply -f pod.yaml -n solar-test + +Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: containers[0] reference "harbor/v2/prod-registry/debian" uses pullPolicy=Never which is not allowed (allowed: Always) +``` + +Because our first rule enforces all registries to use the `always` image pull policy, the pod creation is denied because it uses the `Never` policy. We can either allow the `Never` policy for this specific registry: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + ... + rules: + - enforce: + registries: + + # Enforce PullPolicy "always" for all registries (For Container Images and Volume Images) + - url: ".*" + policy: [ "Always" ] + + # If we are pulling from a harbor registry we want to verify the images for Containers only + - url: "harbor/.*" + policy: [ "Never" ] + + - namespaceSelector: + matchExpressions: + - key: env + operator: In + values: ["prod"] + enforce: + registries: + - url: "harbor/v2/customer-registry/prod-image/.*" + policy: [ "always" ] +``` + +But let's for now also remove the generic rule for all registries and only keep the harbor one: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + ... + rules: + - enforce: + registries: + + # If we are pulling from a harbor registry we want to verify the images for Containers only + - url: "harbor/.*" + policy: [ "Never" ] + + - namespaceSelector: + matchExpressions: + - key: env + operator: In + values: ["prod"] + enforce: + registries: + - url: "harbor/v2/customer-registry/prod-image/.*" + policy: [ "Always" ] +``` + +If we try to apply the pod again, it we will still get an error. The problem is that we are mounting an image volume that is not coming from the allowed harbor registry: + +```bash +Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: volumes[0](volume) reference "quay.io/crio/artifact:v2" is not allowed +``` + +However we would like to only validate the images used in the Pod spec (Containers, InitContainers, EphemeralContainers) and not the ones used in the Volumes. We can achieve this by specifying the [`validation`](#validation) field for the harbor registry: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + ... + rules: + - enforce: + registries: + # If we are pulling from a harbor registry we want to verify the images for Containers only + - url: "harbor/.*" + policy: [ "Never" ] + validation: + - "pod/images" + + - namespaceSelector: + matchExpressions: + - key: env + operator: In + values: ["prod"] + enforce: + registries: + - url: "harbor/v2/customer-registry/prod-image/.*" + policy: [ "Always" ] +``` + +#### Policy + +Define the allowed image pull policies for the specified registry URL. Supported policies are: + + * `Always`: The image is always pulled. + * `IfNotPresent`: The image is pulled only if it is not already present on the node. + * `Never`: The image is never pulled. If the image is not present on the node, the Pod will fail to start. + +This configuration is optional. If no policy is specified, all image pull policies are allowed for the given registry. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + ... + rules: + # Matches all Namespaces and enforces the rule for all of them + - enforce: + registries: + - url: "harbor/v2/customer-registry/.*" + policy: [ "IfNotPresent", "Always" ] +``` + + +#### Validation + +Define on which parts of the Pod the registry policy must be validated. Currently supported validation targets are: + + * `pod/images`: Validate the images used in the Pod spec (For `Containers`, `InitContainers` and `EphemeralContainers`). + * `pod/volumes`: Validate the images used in the Pod `Volumes` ([Read More](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/)) + +**By default, both targets are validated**. You can override this behavior by specifying the `validation` field for each registry: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + ... + rules: + # Matches all Namespaces and enforces the rule for all of them + - enforce: + registries: + - url: "harbor/v2/customer-registry/.*" + validation: + - "pod/images" +``` diff --git a/content/en/docs/whats-new.md b/content/en/docs/whats-new.md index aa434ab..5e3f0d3 100644 --- a/content/en/docs/whats-new.md +++ b/content/en/docs/whats-new.md @@ -5,111 +5,93 @@ description: > weight: 1 --- -## Features - -* Admission Webhooks return warnings for deprecated fields in Capsule resources. You are encouraged to update your resources accordingly. - -* Added `--enable-pprof` flag to enable pprof endpoint for profiling Capsule controller performance. Not recommended for production environments. [Read More](/docs/operating/setup/configuration/#controller-options). - -* Added `--workers` flag to define the `MaxConcurrentReconciles` for relevant controllers [Read More](/docs/operating/setup/configuration/#controller-options). - -* Combined [Capsule Users](/docs/operating/architecture/#capsule-users) Configuration for defining all users and groups which should be considered for Capsule tenancy. This simplifies the configuration and avoids confusion between users and groups. [Read More](/docs/operating/setup/configuration/#users) - -* All namespaced items, which belong to a Capsule Tenant, are now labeled with the Tenant name (eg. `capsule.clastix.io/tenant: solar`). This allows easier filtering and querying of resources belonging to a specific Tenant or Namespace. **Note**: This happens at admission, not in the background. If you want your existing resources to be labeled, you need to reapply them or patch them manually to get the labels added. - -* Delegate Administrators for capsule tenants. Administrators have full control (ownership) over all tenants and their namespaces. [Read More](/docs/operating/architecture/#capsule-administrators) - -* Added Dynamic Resource Allocation (DRA) support. Administrators can now assign allowed DeviceClasses to tenant owners. [Read More](/docs/tenants/enforcement/#dynamic-resource-allocation-dra) - -* All available Classes for a tenant (StorageClasses, GatewayClasses, RuntimeClasses, PriorityClasses, DeviceClasses) are now reported in the Tenant Status. These values can be used by Admission to integrate other resources validation or by external systems for reporting purposes ([Example](/docs/operating/admission-policies/#class-validation)). - -```yaml -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -... -status: - classes: - priority: - - system-cluster-critical - - system-node-critical - runtime: - - customer-containerd - - customer-runu - - customer-virt - - default-runtime - - disallowed - - legacy - storage: - - standard -``` - -* All available Owners for a tenant are now reported in the Tenant Status. This allows external systems to query the Tenant resource for its owners instead of querying the RBAC system. - -```yaml -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -... -status: - owners: - - clusterRoles: - - admin - - capsule-namespace-deleter - kind: Group - name: oidc:org:devops:a - - clusterRoles: - - admin - - capsule-namespace-deleter - - mega-admin - - controller - kind: ServiceAccount - name: system:serviceaccount:capsule:controller - - clusterRoles: - - admin - - capsule-namespace-deleter - kind: User - name: alice -``` - -* Introduction of the `TenantOwner` CRD. [Read More](/docs/tenants/permissions/#tenant-owners) - -```yaml -apiVersion: capsule.clastix.io/v1beta2 -kind: TenantOwner -metadata: - labels: - team: devops - name: devops -spec: - kind: Group - name: "oidc:org:devops:a" - clusterRoles: - - "mega-admin" - - "controller" -``` - -## Fixes - -* Admission Webhooks for namespaces had certain dependencies on the first reconcile of a tenant (namespace being allocated to this tenant). This bug has been fixed and now namespaces are correctly assigned to the tenant (at admission) even if the tenant has not yet been reconciled. - -* The entire core package and admission webhooks have been majorly refactored to improve maintainability and extensibility of Capsule. - -## Documentation - -We have added new documentation for a better experience. See the following Topics: - -* **[Extended Admission Policy Recommendations](/docs/operating/admission-policies/)** -* **[Personas](/docs/operating/admission-policies/)** - -## Ecosystem +## Security 🔒 + +* Advisory [GHSA-qjjm-7j9w-pw72](https://github.com/projectcapsule/capsule/security/advisories/GHSA-qjjm-7j9w-pw72) - **High** - Users can create cluster scoped resources anywhere in the cluster if they are allowed to create `TenantResources`. To immidiatly mitigate this, make sure to use [Impersonation](/docs/replications/tenant/#impersonation) for `TenantResources`. + +* Advisory [GHSA-2ww6-hf35-mfjm](https://github.com/projectcapsule/capsule/security/advisories/GHSA-2ww6-hf35-mfjm) - **Moderate** - Users may hijack namespaces via `namespaces/status` privileges. These privileges must have been explicitly granted by Platform Administrators through RBAC rules to be affected. Requests for the `namespaces/status` subresource are now sent to the Capsule admission webhook as well. + +## Breaking Changes ⚠️ + +* By default, Capsule now uses self-signed cert-manager certificates for its admission webhooks. This used to be an optional setting and has now become the default. If you don't have cert-manager installed, you must explicitly re-enable the Capsule TLS controller as [documented here](/docs/operating/setup/installation/#certificate-management). + +## Features ✨ + +* Complete Renovation of Replications [Read More](/docs/replications/). +* Added `RequiredMetadata` for `Namespaces` created in a `Tenant` [Read More](/docs/tenants/metadata/#requiredmetadata). +* Added rule-based promotions for `ServiceAccounts` in `Tenants` [Read More](/docs/tenants/permissions/#rule-promotion). +* Added Implicit Assignment of `TenantOwner` [Read More](/docs/tenants/permissions/#implicit-tenant-assignment). +* Added Aggregation of `TenantOwner` [Read More](/docs/tenants/permissions/#aggregation). +* Introducing new CRD `RuleStatus` [Read More](/docs/tenants/rules/) +* Introducing `data` field for `Tenants` [Read More](/docs/operating/templating/#data) +* Introducing new OCI Registry enforcement [Read More](/docs/tenants/rules/#registries) +* Added new label `projectcapsule.dev/tenant` which is added for all namespaced resources belonging to a `Tenant` [Read More](/docs/tenants/metadata/#managed). +* Added configuration options for managed RBAC [Read More](docs/operating/setup/configuration/#rbac) +* Added configuration options for Impersonation [Read More](/docs/operating/setup/configuration/#impersonation) +* Added configuration options for Cache invalidation [Read More](/docs/operating/setup/configuration/#cacheinvalidation) +* Added configuration options for Dynamic Admission Webhooks [Read More](/docs/operating/setup/configuration/#admission) +* Added Built-In Installation for Gangplank with the Capsule Proxy [Read More](/docs/proxy/gangplank/) +* `Namespace` admission requests are now only sent to the Capsule admission webhook if the user is considered a capsule user (eg. all ServiceAccounts are considered capsule users). This makes Capsule less disruptive in Outage/Incident scenarios. + +## Fixes 🐛 + +* Fixed `ResourcePool` resource quota calculation when multiple `ResourcePoolClaim`s are present in a namespace but not everything is used. For details, see [ResourcePools bound behavior](/docs/resourcepools/#bound). + +* Improved `matchConditions` for admission webhooks that intercept all namespaced items, to avoid processing subresource requests and Events, improving performance and reducing log noise. + +* `Namespaces` are considered active until the Condition `ContentHasNoFinalizers` is `True`. This means that if a `Namespace` has Finalizers, it will be considered active until the Finalizers are removed. This is a more accurate representation of the state of the `Namespace`, as it can still be active even if it has Finalizers. During this all capsule managed resources are still kept and their deletion is blocked until the Finalizers are removed. + +## Documentation 📚 + +We have added new documentation for a better experience. See the following topics: + +* **[Improved installation overview](/docs/operating/setup/installation/)** +* **[Capsule strict RBAC installation](/docs/operating/setup/installation/#strict-rbac)** + +## Ecosystem 🌐 Newly added documentation to integrate Capsule with other applications: -* [OpenCost](/ecosystem/integrations/opencost/) -* [Headlamp](/ecosystem/integrations/headlamp/) -* [Gangplank](/ecosystem/integrations/gangplank/) -* [Teleport](/ecosystem/integrations/teleport/) -* [Openshift](/docs/operating/setup/openshift/) +* [CoreDNS Plugin](https://github.com/CorentinPtrl/capsule_coredns) (Community Contribution) +* [Argo CD](/ecosystem/integrations/argocd/) +* [Flux CD](/ecosystem/integrations/fluxcd/) + +## Project Updates 💫 + + * Incubating [Sandert (ODC Noord)](https://github.com/sandert-k8s) as Maintainer for documentation and website improvements. + * Incubating [Corentin (CCL Consulting)](https://github.com/CorentinPtrl) as Maintainer as core maintainer. + * Incubating [Lucakuendig (Peak Scale)](https://github.com/lucakuendig) as Community Organizer and Openshift efforts. + +## Roadmap 🗺️ + +In the upcoming releases we are planning to work on the following features: + + * Announcing Capsule Swag (Contribution Rewards) 🎁 + * Capsule: [Custom Resource Quotas](https://github.com/projectcapsule/capsule/issues/1745): A Quota implementation which allows to define custom quota constraints (Enterprise Request). + * Capsule: Porting more Properties to the Namespace Rule Approach. + * Capsule: Adding `transformers` for `Global`/`TenantResources`. + * Capsule: Adding `healthChecks` for `Global`/`TenantResources`. + * Capsule: Using Dynamic Admission to measure Resource Quota Usage at Admission (For Tenant Scope ResourceQuotas and JIT Claiming for ResourcePools) + * Capsule: Introducing Break-The-Glass to allow temporary elevation of permissions for Tenant Owners, with an approval process by Platform Administrators. + * Capsule: Adding custom health checks for ArgoCD to upstream + * Capsule: Adding Generic Implementation for `Global`/`TenantResources`. + * Website: Improving the documentation with more examples and use-cases. + * Capsule-Proxy: Bringing back RBAC reflection to Capsule-Proxy (Generic Namespaced List Permissions) + * Capsule-Proxy: Deprecating ProxySettings on Tenants in favour of GlobalProxySettings + + +## Events 📅 + +* **KubeCon 2026** + * **Project Pavilion**: We will be present again at the [Project Pavilion](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/features-add-ons/project-engagement/#project-pavilion) at KubeCon 2026. The exact schedule has not been announced yet, but we will be hosting a booth and look forward to meeting the community in person again. Feel free to reach out to us if you want to meet us there or have any questions about the project. + + * **Lightning Talk** - Histro Histrov, part of the maintainer team, will be speaking about Capsule at KubeCon 2026 in Amsterdam in a Lightning Talk. [Mark the Session](https://kccnceu2026.sched.com/event/2EFxh/project-lightning-talk-namespace-multi-tenancy-but-all-the-problems-related-to-it-hristo-hristov-maintainer) + +* **Capsule Roundtable Summer 2026 🇨🇭** + * We are planning to host a Capsule Roundtable in Summer 2026 in Switzerland. The exact date and location will be announced soon, but we are looking forward to meeting the community in person and discussing the future of Capsule. If you are interested in attending or want to know more about the event, [feel free to reach out to us](https://peakscale.ch/en/contact/). The event is intended for users to present their use-cases and share their experiences with the project, as well as for us to present the roadmap and gather feedback from the community (Not a sales event). + + +* **CNCF Security Slam 2026** + * Capsule will once again be present at the CNCF and accept contributions from the community to improve the security of the project. [Security Slam 2026](https://securityslam.com/slam26/participating-projects). Recap of the award we received in 2023: + + ![capsule-cncf-secslam](/images/blog/security-slam-2023/receiver.jpg) diff --git a/content/en/ecosystem/integrations/argocd.md b/content/en/ecosystem/integrations/argocd.md new file mode 100644 index 0000000..8c0bbfe --- /dev/null +++ b/content/en/ecosystem/integrations/argocd.md @@ -0,0 +1,233 @@ +--- +title: ArgoCD +description: Capsule Integration with ArgoCD +logo: https://github.com/cncf/artwork/raw/main/projects/argo/icon/color/argo-icon-color.svg +type: single +display: true +integration: true +--- + +## Integration + +## Resource Actions + +You may provide [Custom Resource Actions](https://argo-cd.readthedocs.io/en/stable/operator-manual/resource_actions/) for Capsule specific resources and interactions. + + +### Namespace Resource Actions + +![Namespace Resource Actions](/images/ecosystem/argo-ns-action.png) + +With the following configuration, ArgoCD will show `Cordon` and `Resume` actions for the Namespace resource. The `Cordon` action will set the `projectcapsule.dev/cordoned` label to `true`, while the `Resume` action will set it to `false`. This is only for Namespaces part of a Capsule Tenant. + +```yaml +resource.customizations.actions.Namespace: | + mergeBuiltinActions: true + discovery.lua: | + actions = { + cordon = { + iconClass = "fa fa-solid fa-pause", + disabled = true, + }, + uncordon = { + iconClass = "fa fa-solid fa-play", + disabled = true, + }, + } + + local function has_managed_ownerref() + if obj.metadata == nil or obj.metadata.ownerReferences == nil then + return false + end + + for _, ref in ipairs(obj.metadata.ownerReferences) do + if ref.kind == "Tenant" and ref.apiVersion == "capsule.clastix.io/v1beta2" then + return true + end + end + + return false + end + if not has_managed_ownerref() then + return {} + end + local labels = {} + if obj.metadata ~= nil and obj.metadata.labels ~= nil then + labels = obj.metadata.labels + end + + local cordoned = labels["projectcapsule.dev/cordoned"] == "true" + + if cordoned then + actions["uncordon"].disabled = false + else + actions["cordon"].disabled = false + end + + return actions + + definitions: + - name: cordon + action.lua: | + if obj.metadata == nil then + obj.metadata = {} + end + if obj.metadata.labels == nil then + obj.metadata.labels = {} + end + + obj.metadata.labels["projectcapsule.dev/cordoned"] = "true" + return obj + + - name: uncordon + action.lua: | + if obj.metadata ~= nil and obj.metadata.labels ~= nil then + obj.metadata.labels["projectcapsule.dev/cordoned"] = "false" + end + return obj +``` + +### Tenant Resource Actions + +![Tenant Resource Actions](/images/ecosystem/argo-tenant-action.png) + +With the following configuration, ArgoCD will show `Cordon` and `Resume` actions for the Tenant resource. The `Cordon` action will set the `spec.cordon` field to `true`, while the `Resume` action will set it to `false`. + +```yaml +resource.customizations.actions.capsule.clastix.io_Tenant: | + mergeBuiltinActions: true + discovery.lua: | + actions = {} + actions["cordon"] = { + ["iconClass"] = "fa fa-solid fa-pause", + ["disabled"] = true, + } + actions["uncordon"] = { + ["iconClass"] = "fa fa-solid fa-play", + ["disabled"] = true, + } + + local suspend = false + if obj.spec ~= nil and obj.spec.cordoned ~= nil then + suspend = obj.spec.cordoned + end + + if suspend then + actions["uncordon"]["disabled"] = false + else + actions["cordon"]["disabled"] = false + end + + return actions + + definitions: + - name: cordon + action.lua: | + if obj.spec == nil then + obj.spec = {} + end + obj.spec.cordoned = true + return obj + + - name: uncordon + action.lua: | + if obj.spec ~= nil and obj.spec.cordoned ~= nil and obj.spec.cordoned then + obj.spec.cordoned = false + end + return obj +``` + +## Resource Health + +You may provide [Custom Resource Health](https://argo-cd.readthedocs.io/en/stable/operator-manual/health/) for Capsule specific resources and interactions. + +### Tenant Resource Health + +![Tenant Resource Actions](/images/ecosystem/argo-tenant-health.png) + +`Suspends` a `Tenant` when it's `Cordoned`. Cordoning a Tenant will Cordon/Uncordon all it's Namespaces. + +```yaml +resource.customizations.health.capsule.clastix.io_Tenant: | + hs = {} + if obj.status ~= nil then + if obj.status.conditions ~= nil then + for i, condition in ipairs(obj.status.conditions) do + if condition.type == "Cordoned" and condition.status == "True" then + hs.status = "Suspended" + hs.message = condition.message + return hs + end + end + for i, condition in ipairs(obj.status.conditions) do + if condition.type == "Ready" and condition.status == "False" then + hs.status = "Degraded" + hs.message = condition.message + return hs + end + if condition.type == "Ready" and condition.status == "True" then + hs.status = "Healthy" + hs.message = condition.message + return hs + end + end + end + end + + hs.status = "Progressing" + hs.message = "Waiting for Status" + return hs +``` + +### Namespace Resource Health + +![Namespace Resource Actions](/images/ecosystem/argo-ns-health.png) + +`Suspends` a `Namespace` when it's `Cordoned`. This is only for Namespaces part of a Capsule Tenant. + +```yaml +resource.customizations.health.Namespace: | + hs = {} + local function has_managed_ownerref() + if obj.metadata == nil or obj.metadata.ownerReferences == nil then + return false + end + + for _, ref in ipairs(obj.metadata.ownerReferences) do + if ref.kind == "Tenant" and ref.apiVersion == "capsule.clastix.io/v1beta2" then + return true + end + end + + return false + end + + local labels = {} + if obj.metadata ~= nil and obj.metadata.labels ~= nil then + labels = obj.metadata.labels + end + + local cordoned = labels["projectcapsule.dev/cordoned"] == "true" + + if cordoned and has_managed_ownerref() then + hs.status = "Suspended" + hs.message = "Namespace is cordoned (tenant-managed)" + return hs + end + + if obj.status ~= nil and obj.status.phase ~= nil then + if obj.status.phase == "Active" then + hs.status = "Healthy" + hs.message = "Namespace is Active" + return hs + else + hs.status = "Progressing" + hs.message = "Namespace phase is " .. obj.status.phase + return hs + end + end + + hs.status = "Progressing" + hs.message = "Waiting for Namespace status" + return hs +``` diff --git a/content/en/ecosystem/integrations/envoy-gateway.md b/content/en/ecosystem/integrations/envoy-gateway.md new file mode 100644 index 0000000..c8474af --- /dev/null +++ b/content/en/ecosystem/integrations/envoy-gateway.md @@ -0,0 +1,171 @@ +--- +title: Envoy-Gateway +description: Capsule Integration with Envoy (Gateway API) +logo: https://github.com/cncf/artwork/raw/main/projects/envoy/envoy-gateway/icon/color/envoy-gateway-icon-color.svg +type: single +display: true +integration: true +--- + +There's different ways to use [Gateway API](https://gateway-api.sigs.k8s.io/) in a multi-tenant setup. This guide suggested a strong isolated implementation using the [Envoy Gateway Project](https://gateway.envoyproxy.io/). The Architecture suggested looks something like this: + + +![Namespace Resource Actions](/images/ecosystem/envoy-gateway.drawio.png) + +Each tenant will get it's own `-system` `Namespace`. However that namespace is not managed by the `Tenant` nor part of it. It's the namespace where the platform deploys managed services for each `Tenant`, which are out of bound for `TenantOwners`. + +## Example + +## Gateway + + + + + + + + + +--- +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: EnvoyProxy +metadata: + name: itbs-tenant-{{ $.Values.name }}-gateway + namespace: itbs-tenant-{{ $.Values.name }}-system +spec: + logging: + level: + default: debug + provider: + type: Kubernetes + kubernetes: + envoyDeployment: + replicas: 2 + {{- if $.Values.networking.gateway.loadbalancer }} + envoyService: + loadBalancerIP: {{ $.Values.networking.ingress.loadbalancer }} + {{- end }} +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: itbs-tenant-{{ $.Values.name }}-gateway + namespace: itbs-tenant-{{ $.Values.name }}-system + {{- if $.Values.networking.gateway.issuer.enabled }} + annotations: + cert-manager.io/issuer: itbs-tenant-{{ $.Values.name }}-http + cert-manager.io/private-key-size: "4096" + cert-manager.io/private-key-algorithm: RSA + {{- end }} +spec: + gatewayClassName: {{$.Values.cluster.gateway.classes.platform}} + infrastructure: + parametersRef: + group: gateway.envoyproxy.io + kind: EnvoyProxy + name: itbs-tenant-{{ $.Values.name }}-gateway + listeners: + - name: http-challenge + port: 80 + protocol: HTTP + hostname: "*.{{ $.Values.name }}.{{ $.Values.cluster.name }}.{{ $.Values.infrastructure.dns.zone }}" + allowedRoutes: # Only this tenant's capsule namespaces can attach routes to this listener + namespaces: + from: Selector + selector: + matchLabels: + tenant.itbs.ch/tenant: "{{ $.Values.name }}" + + {{- if $.Values.metrics.enabled }} + - name: https-alertmanager + protocol: HTTPS + port: 443 + hostname: "alertmanager.{{ $.Values.name }}.{{ .Values.cluster.name }}.{{ .Values.infrastructure.dns.zone }}" + tls: + mode: Terminate + certificateRefs: + - group: '' + kind: Secret + name: alertmanager-tls + allowedRoutes: + namespaces: + from: Selector + selector: + matchLabels: + tenant.itbs.ch/tenant-system: "{{ $.Values.name }}" + - name: https-prometheus + protocol: HTTPS + port: 443 + hostname: "prometheus.{{ $.Values.name }}.{{ .Values.cluster.name }}.{{ .Values.infrastructure.dns.zone }}" + tls: + mode: Terminate + certificateRefs: + - group: '' + kind: Secret + name: prometheus-tls + allowedRoutes: + namespaces: + from: Selector + selector: + matchLabels: + tenant.itbs.ch/tenant-system: "{{ $.Values.name }}" + {{- end }} + {{- if $.Values.grafana.enabled }} + - name: https-grafana + protocol: HTTPS + port: 443 + hostname: "grafana.{{ $.Values.name }}.{{ .Values.cluster.name }}.{{ .Values.infrastructure.dns.zone }}" + tls: + mode: Terminate + certificateRefs: + - group: '' + kind: Secret + name: grafana-tls + allowedRoutes: + namespaces: + from: Selector + selector: + matchLabels: + tenant.itbs.ch/tenant-system: "{{ $.Values.name }}" + {{- end }} + + + +## EnvoyProxy + + +## Certificate Management + +If we additionally would like to do Certificate Management via [cert-manager](https://cert-manager.io/docs/) in combination with [ACME HTTP-01 challenges](https://cert-manager.io/docs/configuration/acme/http01/) we probably want to provide the users with a `ClusterIssuer` per `Tenant`: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-acme-issuer +spec: + scope: Tenant + resources: + - rawItems: + - apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: {{tenant.name}}-acme-http + namespace: {{tenant.name}}-system + spec: + acme: + email: platform@email.com + server: https://acme-staging-v02.api.letsencrypt.org/directory + privateKeySecretRef: + name: cert-letsencrypt-staging + solvers: + - http01: + gatewayHTTPRoute: + parentRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: {{tenant.name}}-gateway + namespace: {{tenant.name}}-system + sectionName: http-challenge +``` + diff --git a/content/en/ecosystem/integrations/eso.md b/content/en/ecosystem/integrations/eso.md new file mode 100644 index 0000000..0fedaab --- /dev/null +++ b/content/en/ecosystem/integrations/eso.md @@ -0,0 +1,51 @@ +--- +title: External Secrets Operator +description: Capsule Integration with External Secrets Operator +logo: https://github.com/cncf/artwork/raw/main/projects/external-secrets-operator/icon/color/eso-icon-color.svg +type: single +display: true +integration: true +--- + +With [External Secrets Operator](https://external-secrets.io/latest/) it's possible to delegate Secrets Management to an external system while keeping the actual management of the secrets within Kubernetes. This guide provides a simple automation example with [External Secrets Operator](https://external-secrets.io/latest/). Before starting, you might want to explore the existing documentation regarding multi-tenancy: + + * [https://external-secrets.io/latest/guides/multi-tenancy/](https://external-secrets.io/latest/guides/multi-tenancy/) + +## Secure ClusterSecretStores + +If you have any `ClusterSecretStores`, which are not intended to be used by `Tenants`, you must make sure `Tenants` can not reference the `ClusterSecretStore`. You can achieve this by unselecting all `Tenant` `Namespaces` like so: + +```yaml +--- +apiVersion: external-secrets.io/v1 +kind: ClusterSecretStore +metadata: + name: platform-vault +spec: + conditions: + - namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant # Forbid the use of this platform keyvault by tenants + operator: DoesNotExist + provider: + azurekv: + tenantId: {TENANT} + vaultUrl: {VAULT} + authSecretRef: + clientId: + name: external-secrets-secret + key: azure.clientID + namespace: external-secrets + clientSecret: + name: external-secrets-secret + key: azure.clientSecret + namespace: external-secrets +``` + +## ClusterSecretStores + + + + + + diff --git a/content/en/ecosystem/integrations/gangplank.md b/content/en/ecosystem/integrations/gangplank.md deleted file mode 100644 index bfe56ab..0000000 --- a/content/en/ecosystem/integrations/gangplank.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -title: Gangplank -description: Capsule Integration with Gangplank -logo: https://avatars.githubusercontent.com/u/29403644?s=280&v=4 -type: single -display: true -integration: true ---- - -[Gangplank](https://github.com/sighupio/gangplank) is a web application that allows users to authenticate with an OIDC provider and configure their kubectl configuration file with the [OpenID Connect Tokens](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens). Gangplank is based on [Gangway](https://github.com/vmware-archive/gangway), which is no longer maintained. - -## Prerequisites - -1. You will need a running [Capsule Proxy](/docs/proxy/) instance. -2. For Authentication you will need a Confidential OIDC client configured in your OIDC provider, such as [Keycloak](https://www.keycloak.org/), [Dex](https://dexidp.io/), or [Google Cloud Identity](https://docs.github.com/en/actions/how-tos/secure-your-work/security-harden-deployments/oidc-in-google-cloud-platform). By default the Kubernetes API only validates tokens against a Public OIDC client, so you will need to configure your OIDC provider to allow the Gangplank client to issue tokens. You must make use of the Kubernetes Authentication Configuration, which allows to define multiple audiences (clients). This way we can issue tokens for a gangplank client, which is Confidential, and a kubernetes client, which is Public. The Kubernetes API will validate the tokens against both clients. The Config might look like this: - -```yaml -apiVersion: apiserver.config.k8s.io/v1beta1 -kind: AuthenticationConfiguration -jwt: -- issuer: - url: https://keycloak/realms/realm-name - audiences: - - kubernetes - - gangplank - audienceMatchPolicy: MatchAny # This one is important - claimMappings: - username: - claim: 'email' - prefix: "" - groups: - claim: 'groups' - prefix: "" -``` - -[Read More](/docs/operating/authentication/#configuring-kubernetes-api-server) - -## Integration - -To install Gangplank, you can use the Helm chart provided in the [Gangplank repository](https://github.com/sighupio/gangplank/blob/main/deployments/helm/values.yaml) or use your own custom values file. The following Environment Variables are required: - -* `GANGPLANK_CONFIG_AUTHORIZE_URL`: `https://keycloak/realms/realm-name/protocol/openid-connect/auth` -* `GANGPLANK_CONFIG_TOKEN_URL`: `https://keycloak/realms/realm-name/protocol/openid-connect/token` -* `GANGPLANK_CONFIG_REDIRECT_URL`: `https://gangplank.example.com/callback` -* `GANGPLANK_CONFIG_CLIENT_ID`: `gangplank` -* `GANGPLANK_CONFIG_CLIENT_SECRET`: `` -* `GANGPLANK_CONFIG_USERNAME_CLAIM`: The JWT claim to use as the username. (we use `email` in the authentication config above, so this should also be `email`) -* `GANGPLANK_CONFIG_APISERVER_URL`: The URL **Capsule Proxy Ingress**. Since the users probably want to access the Kubernetes API from outside the cluster, you should use the Capsule Proxy Ingress URL here. - -When using the Helm chart, you can set these values in the `values.yaml` file: - -```yaml -config: - clusterName: "tenant-cluster" - apiServerURL: "https://capsule-proxy.company.com:443" - scopes: ["openid", "profile", "email", "groups", "offline_access"] - redirectURL: "https://gangplank.company.com/callback" - usernameClaim: "email" - clientID: "gangplank" - authorizeURL: "https://keycloak/realms/realm-name/protocol/openid-connect/auth" - tokenURL: "https://keycloak/realms/realm-name/protocol/openid-connect/token" - -# Mount The Client Secret as Environment Variables (GANGPLANK_CONFIG_CLIENT_SECRET) -envFrom: -- secretRef: - name: gangplank-secrets -``` - -Now the only thing left to do is to change the CA certificate which is provided. By default the CA certificate is set to the Kubernetes API server CA certificate, which is not valid for the Capsule Proxy Ingress. For this we can simply override the CA certificate in the Helm chart. You can do this by creating a Kubernetes Secret with the CA certificate and mounting it as a volume in the Gangplank deployment. - -```yaml -volumeMounts: - - mountPath: /var/run/secrets/kubernetes.io/serviceaccount - name: token-ca -volumes: - - name: token-ca - projected: - sources: - - serviceAccountToken: - path: token - - secret: - name: proxy-ingress-tls - items: - - key: tls.crt - path: ca.crt -``` - -**Note**: In this example we used the `tls.crt` key of the `proxy-ingress-tls` secret. This is a classic [Cert-Manager](https://cert-manager.io/) TLS secret, which contains only the Certificate and Key for the Capsule Proxy Ingress. However the Certificate contains the CA certificate as well (Certificate Chain), so we can use it to verify the Capsule Proxy Ingress. If you use a different secret, make sure to adjust the key accordingly. - -If that's not possible you can also set the CA certificate as an environment variable: - -```yaml -config: - clusterCAPath: "/capsule-proxy/ca.crt" -volumeMounts: - - mountPath: /capsule-proxy/ - name: token-ca -volumes: - - name: token-ca - projected: - sources: - - secret: - name: proxy-ingress-tls - items: - - key: tls.crt - path: ca.crt - - - - - - - - - - - diff --git a/content/en/ecosystem/integrations/harbor.md b/content/en/ecosystem/integrations/harbor.md new file mode 100644 index 0000000..973732d --- /dev/null +++ b/content/en/ecosystem/integrations/harbor.md @@ -0,0 +1,98 @@ +--- +title: Envoy-Gateway +description: Capsule Integration with Harbor +logo: https://github.com/cncf/artwork/raw/main/projects/envoy/envoy-gateway/icon/color/envoy-gateway-icon-color.svg +type: single +display: true +integration: true +--- + +There's different ways to use [Gateway API](https://gateway-api.sigs.k8s.io/) in a multi-tenant setup. This guide suggested a strong isolated implementation using the [Envoy Gateway Project](https://gateway.envoyproxy.io/). The Architecture suggested looks something like this: + +![Namespace Resource Actions](/images/ecosystem/envoy-gateway.drawio.png) + +Each tenant will get it's own `-system` `Namespace`. However that namespace is not managed by the `Tenant` nor part of it. It's the namespace where the platform deploys managed services for each `Tenant`, which are out of bound for `TenantOwners`. + +## Registry Overwrite + +## Management (Crossplane) + +The following example shows how you could automate the management of Harbor based Tenants. This assumes you provide a single harbor instance where all Tenants host their Harbor Projects. However this approach requires [Crossplane](https://www.crossplane.io/) in combindation with the [community provider for Harbor](https://github.com/globallogicuki/provider-harbor). + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-harbor-project +spec: + scope: Tenant + resources: + - generators: + - template: | + --- + apiVersion: project.harbor.crossplane.io/v1alpha1 + kind: Project + metadata: + name: {{$.tenant.metadata.name}} + labels: + projectcapsule.dev/tenant: {{$.tenant.metadata.name}} + spec: + forProvider: + autoSbomGeneration: true + enableContentTrust: true + enableContentTrustCosign: false + name: {{$.tenant.metadata.name}} + public: false + vulnerabilityScanning: true + {{- with $.tenant.data.registryStorageQuota }} + storageQuota: 10 + {{- end }} + --- + apiVersion: project.harbor.crossplane.io/v1alpha1 + kind: RetentionPolicy + metadata: + name: {{$.tenant.metadata.name}} + spec: + forProvider: + rule: + - nDaysSinceLastPull: 5 + repoMatching: '**' + tagMatching: latest + - nDaysSinceLastPush: 10 + repoMatching: '**' + tagMatching: '{latest,snapshot}' + schedule: Daily + scopeSelector: + matchLabels: + projectcapsule.dev/tenant: {{$.tenant.metadata.name}} + - generators: + - template: | + {{- range $.tenant.status.owners }} + {{- if eq .kind "User" }} + --- + apiVersion: project.harbor.crossplane.io/v1alpha1 + kind: MemberGroup + metadata: + name: {{$.tenant.metadata.name}}-owner-group-{{.name}} + spec: + forProvider: + groupName: {{.name}} + projectIdSelector: + matchLabels: + projectcapsule.dev/tenant: {{$.tenant.metadata.name}} + role: projectadmin + type: oidc + {{- elseif eq .kind "User" }} + --- + apiVersion: project.harbor.crossplane.io/v1alpha1 + kind: MemberUser + metadata: + name: {{$.tenant.metadata.name}}-owner-user-{{.name}} + spec: + forProvider: + projectIdSelector: + matchLabels: + projectcapsule.dev/tenant: {{$.tenant.metadata.name}} + role: projectadmin + userName: {{.name}} + {{- end }} \ No newline at end of file diff --git a/data/resources.yaml b/data/resources.yaml index 9111e59..dbcfc60 100644 --- a/data/resources.yaml +++ b/data/resources.yaml @@ -1,4 +1,22 @@ resources: + - title: "REX Ubisoft - Quand et comment partager un cluster : retour d'expérience sur Capsule chez Ubisoft" + youtube: "NAx2s_o1iuE" + date: "2026-02-25" + type: "video" + tags: + - customer-journey + - title: "REX DINUM -Les ingrédients multitenancy et authentification pour une distribution k8s open-source" + youtube: "zWayprWXWVg" + date: "2026-02-25" + type: "video" + tags: + - customer-journey + - title: "REX Renault - Kubernetes as a Service : sécurité, innovation et self-service à grande échelle" + youtube: "F8x6DBeNeqg" + date: "2026-02-25" + type: "video" + tags: + - customer-journey - title: "The State of Multi-tenancy in Kubernetes by LoftLabs" url: https://www.linkedin.com/events/7297698298100936704/comments/ date: "2025-02-27" diff --git a/diagrams/tenant-replication.drawio b/diagrams/tenant-replication.drawio new file mode 100644 index 0000000..1079064 --- /dev/null +++ b/diagrams/tenant-replication.drawio @@ -0,0 +1,360 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/static/images/content/replication-globaltenantresource.png b/static/images/content/replication-globaltenantresource.png new file mode 100644 index 0000000..f33aa06 Binary files /dev/null and b/static/images/content/replication-globaltenantresource.png differ diff --git a/static/images/content/replication-tenantresource.png b/static/images/content/replication-tenantresource.png new file mode 100644 index 0000000..79fd999 Binary files /dev/null and b/static/images/content/replication-tenantresource.png differ diff --git a/static/images/ecosystem/argo-ns-action.png b/static/images/ecosystem/argo-ns-action.png new file mode 100644 index 0000000..713e165 Binary files /dev/null and b/static/images/ecosystem/argo-ns-action.png differ diff --git a/static/images/ecosystem/argo-ns-health.png b/static/images/ecosystem/argo-ns-health.png new file mode 100644 index 0000000..49449ed Binary files /dev/null and b/static/images/ecosystem/argo-ns-health.png differ diff --git a/static/images/ecosystem/argo-tenant-action.png b/static/images/ecosystem/argo-tenant-action.png new file mode 100644 index 0000000..a9a39e2 Binary files /dev/null and b/static/images/ecosystem/argo-tenant-action.png differ diff --git a/static/images/ecosystem/argo-tenant-health.png b/static/images/ecosystem/argo-tenant-health.png new file mode 100644 index 0000000..836d7d5 Binary files /dev/null and b/static/images/ecosystem/argo-tenant-health.png differ diff --git a/static/images/ecosystem/envoy-gateway.drawio.png b/static/images/ecosystem/envoy-gateway.drawio.png new file mode 100644 index 0000000..4902e56 Binary files /dev/null and b/static/images/ecosystem/envoy-gateway.drawio.png differ