From 70593b717a968d6194aafb9a82ccf405fe415748 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Wed, 12 Mar 2025 11:43:54 +0100 Subject: [PATCH 01/18] chore(MEP-16): initial draft metal-stack/cluster-api-provider-metal-stack#39 --- .../src/development/proposals/MEP16/README.md | 88 +++++++++++++++++++ docs/src/development/proposals/index.md | 1 + 2 files changed, 89 insertions(+) create mode 100644 docs/src/development/proposals/MEP16/README.md diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md new file mode 100644 index 0000000000..9ee340ae5f --- /dev/null +++ b/docs/src/development/proposals/MEP16/README.md @@ -0,0 +1,88 @@ +# Firewall Support for Cluster API Provider + +Currently the creation and management of firewalls is out of scope for the [Cluster API Provider metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack), or in short capms. In practice this requires operators to create the firewall and therefore also the node network before generating the cluster. +When either the firewall rules, the firewall image or the firewall machine size change, this requires the operator to manually roll the firewall by creating a new one and deleting the old firewall. + +To gain a production ready implementation for Cluster API to automatically manage the deployment of firewalls, it makes sense to build on top of prior art. +In case of Gardener on metal-stack the [Firewall Controller Manager](https://github.com/metal-stack/firewall-controller-manager), or short fcm, is used. + +## Overview + +The capms controller manager should now create the node network if needed. And when a firewall template exists for the cluster, a firewall deployment should be created and updated on every change. + +The fcm should now observe all firewall deployments across all namespaces. It then creates firewall sets and later firewalls. It handles and implement rolling updates. + +## Implementation + +During reconciliation the capms controller manager has a look at `MetalStackCluster.Spec.NodeNetworkID`. If it is not yet set, the node network should created and patched back into the spec. Then the manager should move to the `MetalStackCluster.Spec.FirewallTemplate`. When set, it should create or update the matching `FirewallDeployment` according to the nested data like the image, networks, size or the `FirewallDeployment.Spec.Template.InitialRuleSet`. + +From here on the fcm will take over to reconcile all `FirewallDeployment`s. Currently it only watches a specific namespace and needs to be adapted to be capable of operating on all namespaces instead. Also initial firewall rule sets need to be implemented. + +When the operator installs Cluster API including capms into their cluster, `clusterctl init --infrastructure metal-stack` needs to include the fcm custom resources and optionally its controller deployment to be able to reconcile firewalls. + +When generating a new cluster using `clusterctl generate cluster --infrastructure metal-stack cluster-name`, this should also generate a firewall entry including some basic firewall rules. + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 +kind: MetalStackCluster +metadata: + name: cluster-name + namespace: default +spec: + projectID: some-project-id + partition: partition + nodeNetworkID: node-network-id # now optional if firewall given + firewallTemplate: + size: machine-size + image: firewall-ubuntu + networks: [] # the node-network-id will be appended automatically + initialRuleSet: + ingress: + - comment: allow incoming https + protocol: tcp + ports: [443] + from: [0.0.0.0/0] + egress: + - comment: allow outgoing https + protocol: tcp + ports: [443] + to: [0.0.0.0/0] + # additional entries for dns, ssh and ntp required +``` + +Cluster API allows to move the resource and therefore the control over the bootstrapped workload cluster. As part of this process `clusterctl` marks the cluster as paused. Providers like capms need to obey the pause and may not reconcile during this time period to avoid loss of infrastructure and data. + +When using the fcm in this context, it should also obey this rule. But in order to keep the fcm separated from cluster api internals, we'd like to propose the `firewall.metal-stack.io/paused` annotation to pause the reconciliation. Managing this pause annotation on the `FirewallDeployment` is in the responsibility of the capms controller manager. + +This annoation still needs to be propagated to nested resources like `FirewallSet` and `Firewall`. + +Furthermore Cluster API also requires to have an unbroken chain of owner refrences. In practice this means `Firewall` is owned by `FirewallSet` which is in turn owned by `FirewallDeployment`. This is a [Kubernetes feature](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/) and good practice anyways. + +## Affected Components + +For some working items, we already created proof of concept PRs, which might need to be splitted up and are not yet production ready. + +- Firewall Controller Manager + - must be configured to watch all namespaces for `FirewallDeployment` [fcm#66](https://github.com/metal-stack/firewall-controller-manager/pull/66) + - initial firewall rules needs to be implemented [fcm#64](https://github.com/metal-stack/firewall-controller-manager/pull/64) + - the annotations of the `FirewallDeployment` need to be propagated down to the `FirewallSet` and `Firewall` + - needs to introduce and implement the `firewall.metal-stack.io/paused` annotation + - set owner references to `FirewallSet` and `Firewall` +- Cluster API Controller Manager + - the `FirewallDeployment` pause annotation needs to be managed + - the `FirewallDeployment` resource should be created if template set [capms#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) + - management of SSH keys for machines and firewalls [capms#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) + - cluster-template + - needs to install CRDs for the FCM in order to be respected during moves [capms#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) + - eventually also install the FCM itself + - `MetalStackCluster` resource defintion + - add `MetalStackCluster.Spec.FirewallTemplate` + - make `Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given + +## Caveats and Organizational Implications + +When the cluster is pivoted and reconciles its own firewall, a malfunctioning firewall prevents the cluster from self-healing and requires manual intervention by creating a new firewall. This is an inherent problem of the cluster-api approach. It can be circumvented by using an extra cluster to manage workload clusters. + +In the current form of this approach firewalls and therefore the firewall egress and ingress rules are managed by the cluster operators that manage the cluster-api resources. +Hence it will not be possible to gain a fine-grained control over every cluster operator's choices from a central ruleset at the level of metal-stack firewalls. +In case this control surfaces as a requirement, it would need to be implemented in a firewall external to metal-stack. diff --git a/docs/src/development/proposals/index.md b/docs/src/development/proposals/index.md index 573d61d29f..8ccef1b907 100644 --- a/docs/src/development/proposals/index.md +++ b/docs/src/development/proposals/index.md @@ -32,3 +32,4 @@ Once a proposal was accepted, an issue should be raised and the implementation s | [MEP-12](MEP12/README.md) | Rack Spreading | `Completed` | | [MEP-13](MEP13/README.md) | IPv6 | `Completed` | | [MEP-14](MEP14/README.md) | Independence from external sources | `Completed` | +| [MEP-16](MEP16/README.md) | Firewall Support for Cluster API Provider | `In Discussion` | From 886936d71a0381e24c6aae6d1faf2c5b27862c42 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Wed, 12 Mar 2025 11:49:38 +0100 Subject: [PATCH 02/18] fix: thanks typos --- docs/src/development/proposals/MEP16/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 9ee340ae5f..8035aa7ca8 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -54,13 +54,13 @@ Cluster API allows to move the resource and therefore the control over the boots When using the fcm in this context, it should also obey this rule. But in order to keep the fcm separated from cluster api internals, we'd like to propose the `firewall.metal-stack.io/paused` annotation to pause the reconciliation. Managing this pause annotation on the `FirewallDeployment` is in the responsibility of the capms controller manager. -This annoation still needs to be propagated to nested resources like `FirewallSet` and `Firewall`. +This annotation still needs to be propagated to nested resources like `FirewallSet` and `Firewall`. -Furthermore Cluster API also requires to have an unbroken chain of owner refrences. In practice this means `Firewall` is owned by `FirewallSet` which is in turn owned by `FirewallDeployment`. This is a [Kubernetes feature](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/) and good practice anyways. +Furthermore Cluster API also requires to have an unbroken chain of owner references. In practice this means `Firewall` is owned by `FirewallSet` which is in turn owned by `FirewallDeployment`. This is a [Kubernetes feature](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/) and good practice anyways. ## Affected Components -For some working items, we already created proof of concept PRs, which might need to be splitted up and are not yet production ready. +For some working items, we already created proof of concept PRs, which might need to be split up and are not yet production ready. - Firewall Controller Manager - must be configured to watch all namespaces for `FirewallDeployment` [fcm#66](https://github.com/metal-stack/firewall-controller-manager/pull/66) @@ -75,7 +75,7 @@ For some working items, we already created proof of concept PRs, which might nee - cluster-template - needs to install CRDs for the FCM in order to be respected during moves [capms#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) - eventually also install the FCM itself - - `MetalStackCluster` resource defintion + - `MetalStackCluster` resource definition - add `MetalStackCluster.Spec.FirewallTemplate` - make `Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given From b01267d163e26f83fdbc97339fcfc7255781b2e4 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Wed, 12 Mar 2025 14:49:53 +0100 Subject: [PATCH 03/18] docs(MEP-16): architecture view --- docs/src/development/proposals/MEP16/README.md | 2 ++ .../proposals/MEP16/firewall-for-capms-overview.svg | 4 ++++ 2 files changed, 6 insertions(+) create mode 100644 docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 8035aa7ca8..b12730fd1a 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -8,6 +8,8 @@ In case of Gardener on metal-stack the [Firewall Controller Manager](https://git ## Overview +![architectural overvier](firewall-for-capms-overview.svg) + The capms controller manager should now create the node network if needed. And when a firewall template exists for the cluster, a firewall deployment should be created and updated on every change. The fcm should now observe all firewall deployments across all namespaces. It then creates firewall sets and later firewalls. It handles and implement rolling updates. diff --git a/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg new file mode 100644 index 0000000000..07fe7999b4 --- /dev/null +++ b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg @@ -0,0 +1,4 @@ + + + +
Handles traffic
Handles traffic
Firewall
Firewall
Firewall Controller
Firewall Controller
Workload Cluster
Workload Cluster
Configures
Configures
Management Cluster
Management Cluster
Cluster API Provider metal-stack
Cluster API Provider...
MetalStackCluster
MetalStackCluster
Firewall Deployment
Firewall Deployment
Firewall CRD
Firewall CRD
Firewall Controller Manager
Firewall Controller...
Pushes Configuration
Pushes Configuration
Configures
Configures
Operator
Opera... \ No newline at end of file From 11d20925c93f3d16ee18901c6b3814f24ad48121 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Thu, 13 Mar 2025 09:09:56 +0100 Subject: [PATCH 04/18] style: slightly tweaks --- .../src/development/proposals/MEP16/README.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index b12730fd1a..18becc1054 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -1,7 +1,7 @@ # Firewall Support for Cluster API Provider -Currently the creation and management of firewalls is out of scope for the [Cluster API Provider metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack), or in short capms. In practice this requires operators to create the firewall and therefore also the node network before generating the cluster. -When either the firewall rules, the firewall image or the firewall machine size change, this requires the operator to manually roll the firewall by creating a new one and deleting the old firewall. +Currently the creation and management of firewalls is out of scope for the [Cluster API Provider metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack), or in short CAPMS. In practice this requires operators to create the firewall and therefore also the node network before generating the cluster. +When either the firewall rules, the firewall image or the firewall machine size change, the current appraoch also requires the operator to manually roll the firewall by creating a new one and deleting the old firewall. To gain a production ready implementation for Cluster API to automatically manage the deployment of firewalls, it makes sense to build on top of prior art. In case of Gardener on metal-stack the [Firewall Controller Manager](https://github.com/metal-stack/firewall-controller-manager), or short fcm, is used. @@ -10,17 +10,17 @@ In case of Gardener on metal-stack the [Firewall Controller Manager](https://git ![architectural overvier](firewall-for-capms-overview.svg) -The capms controller manager should now create the node network if needed. And when a firewall template exists for the cluster, a firewall deployment should be created and updated on every change. +The CAPMS controller manager should now create the node network if needed. And when a firewall template exists for the cluster, a firewall deployment should be created and updated on every change. -The fcm should now observe all firewall deployments across all namespaces. It then creates firewall sets and later firewalls. It handles and implement rolling updates. +The fcm should now observe all firewall deployments across all namespaces. It then creates firewall sets and later firewalls. It already handles and implements rolling updates. ## Implementation -During reconciliation the capms controller manager has a look at `MetalStackCluster.Spec.NodeNetworkID`. If it is not yet set, the node network should created and patched back into the spec. Then the manager should move to the `MetalStackCluster.Spec.FirewallTemplate`. When set, it should create or update the matching `FirewallDeployment` according to the nested data like the image, networks, size or the `FirewallDeployment.Spec.Template.InitialRuleSet`. +During reconciliation the CAPMS controller manager has a look at `MetalStackCluster.Spec.NodeNetworkID`. If it is not yet set, the node network should created and patched back into the spec. Then the manager should move to the `MetalStackCluster.Spec.FirewallTemplate`. When set, it should create or update the matching `FirewallDeployment` according to the nested data like the image, networks, size or the `FirewallDeployment.Spec.Template.InitialRuleSet`. From here on the fcm will take over to reconcile all `FirewallDeployment`s. Currently it only watches a specific namespace and needs to be adapted to be capable of operating on all namespaces instead. Also initial firewall rule sets need to be implemented. -When the operator installs Cluster API including capms into their cluster, `clusterctl init --infrastructure metal-stack` needs to include the fcm custom resources and optionally its controller deployment to be able to reconcile firewalls. +When the operator installs Cluster API including CAPMS into their cluster, `clusterctl init --infrastructure metal-stack` needs to include the fcm custom resources and optionally its controller deployment to be able to reconcile firewalls. When generating a new cluster using `clusterctl generate cluster --infrastructure metal-stack cluster-name`, this should also generate a firewall entry including some basic firewall rules. @@ -52,9 +52,9 @@ spec: # additional entries for dns, ssh and ntp required ``` -Cluster API allows to move the resource and therefore the control over the bootstrapped workload cluster. As part of this process `clusterctl` marks the cluster as paused. Providers like capms need to obey the pause and may not reconcile during this time period to avoid loss of infrastructure and data. +Cluster API allows to move the resource and therefore the control over the bootstrapped workload cluster. As part of this process `clusterctl` marks the cluster as paused. Providers like CAPMS need to obey the pause and may not reconcile during this time period to avoid loss of infrastructure and data. -When using the fcm in this context, it should also obey this rule. But in order to keep the fcm separated from cluster api internals, we'd like to propose the `firewall.metal-stack.io/paused` annotation to pause the reconciliation. Managing this pause annotation on the `FirewallDeployment` is in the responsibility of the capms controller manager. +When using the fcm in this context, it should also obey this rule. But in order to keep the fcm separated from cluster api internals, we'd like to propose the `firewall.metal-stack.io/paused` annotation to pause the reconciliation. Managing this pause annotation on the `FirewallDeployment` is in the responsibility of the CAPMS controller manager. This annotation still needs to be propagated to nested resources like `FirewallSet` and `Firewall`. @@ -72,10 +72,10 @@ For some working items, we already created proof of concept PRs, which might nee - set owner references to `FirewallSet` and `Firewall` - Cluster API Controller Manager - the `FirewallDeployment` pause annotation needs to be managed - - the `FirewallDeployment` resource should be created if template set [capms#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) - - management of SSH keys for machines and firewalls [capms#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) + - the `FirewallDeployment` resource should be created if template set [CAPMS#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) + - management of SSH keys for machines and firewalls [CAPMS#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) - cluster-template - - needs to install CRDs for the FCM in order to be respected during moves [capms#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) + - needs to install CRDs for the FCM in order to be respected during moves [CAPMS#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) - eventually also install the FCM itself - `MetalStackCluster` resource definition - add `MetalStackCluster.Spec.FirewallTemplate` From e4e953793ea531fe5c3e9aa2111c013ebe12d321 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Thu, 13 Mar 2025 09:11:05 +0100 Subject: [PATCH 05/18] fix: typos --- docs/src/development/proposals/MEP16/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 18becc1054..b043cf2e7d 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -1,7 +1,7 @@ # Firewall Support for Cluster API Provider Currently the creation and management of firewalls is out of scope for the [Cluster API Provider metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack), or in short CAPMS. In practice this requires operators to create the firewall and therefore also the node network before generating the cluster. -When either the firewall rules, the firewall image or the firewall machine size change, the current appraoch also requires the operator to manually roll the firewall by creating a new one and deleting the old firewall. +When either the firewall rules, the firewall image or the firewall machine size change, the current approach also requires the operator to manually roll the firewall by creating a new one and deleting the old firewall. To gain a production ready implementation for Cluster API to automatically manage the deployment of firewalls, it makes sense to build on top of prior art. In case of Gardener on metal-stack the [Firewall Controller Manager](https://github.com/metal-stack/firewall-controller-manager), or short fcm, is used. From 45e929823e20872a1790e8ee8d148290ef1c4d41 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Thu, 13 Mar 2025 15:41:53 +0100 Subject: [PATCH 06/18] docs(MEP-16): redesign Co-Authored-By: @Gerrit91 --- .../src/development/proposals/MEP16/README.md | 211 ++++++++++++------ 1 file changed, 146 insertions(+), 65 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index b043cf2e7d..b81543c777 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -1,90 +1,171 @@ -# Firewall Support for Cluster API Provider +# metal-api as an Alternative Configuration Source for the firewall-controller -Currently the creation and management of firewalls is out of scope for the [Cluster API Provider metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack), or in short CAPMS. In practice this requires operators to create the firewall and therefore also the node network before generating the cluster. -When either the firewall rules, the firewall image or the firewall machine size change, the current approach also requires the operator to manually roll the firewall by creating a new one and deleting the old firewall. +In the current situation, a firewall as provisioned by metal-stack is a fully immutable entity. Any modifications on the firewall like changing the firewall ruleset must be done _somehow_ by the user – the metal-api and hence the metal-stack is not aware of its current state. -To gain a production ready implementation for Cluster API to automatically manage the deployment of firewalls, it makes sense to build on top of prior art. -In case of Gardener on metal-stack the [Firewall Controller Manager](https://github.com/metal-stack/firewall-controller-manager), or short fcm, is used. +As part of our [integration with the Gardener project](https://docs.metal-stack.io/stable/overview/kubernetes/#Gardener) we offer a solution called the [firewall-controller](https://github.com/metal-stack/firewall-controller), which is part of our [firewall OS images](https://github.com/metal-stack/metal-images/blob/6318a624861b18a559a9d37299bca5f760eef524/firewall/Dockerfile#L57-L58) and addresses shortcomings of the firewall resource's immutability, which would otherwise be completely impractible to work with. The firewall-controller crashes infinitely if it is not properly configured through the userdata when using the firewall image of metal-stack. -## Overview +The firewall-controller approach is tightly coupled to the Gardener and it requires the administrator of the Gardener installation to pass a shoot and a seed kubeconfig through machine userdata when creating the firewall. How this userdata has to look like is not documented and is just part of another project called the [firewall-controller-manager](https://github.com/metal-stack/firewall-controller-manager), which task is to orchestrate rolling updates of firewall machines in a way that network traffic interruption is minimal when updating a firewall or applying a change to an immutable firewall configuration. -![architectural overvier](firewall-for-capms-overview.svg) +In general, a firewall entity in metal-stack has similarities to the machine entity but it has a fundamental difference: A user gains ownership over a machine after provisioning. They can access it through SSH, modify it at will and this is completely wanted. For firewalls, however, we do not want a user to access the provisioned firewall as the firewall is a privileged part of the infrastructure with access to the underlay network. The underlay can not be tampered with at any given point in time by a user as it can destroy the entire network traffic flow inside a metal-stack partition. -The CAPMS controller manager should now create the node network if needed. And when a firewall template exists for the cluster, a firewall deployment should be created and updated on every change. +For this reason, we have a gap in the metal-stack project in terms of a missing solution for people who do not rely on the Gardener integration. We are basically leaving a user with the option to implement an orchestrated recreation of every possible change on the firewall to minimize traffic interruption for the machines sitting behind the firewall or re-implement the firewall-controller to how they want to use it for their use-case. Also we do not have a clear distinction in the API between user and landscape operator for firewalls. If a user would allocate firewall it is also possible for the user to inject his own SSH keys and access the firewall and tamper with the underlay network. -The fcm should now observe all firewall deployments across all namespaces. It then creates firewall sets and later firewalls. It already handles and implements rolling updates. +Parts of these problems are probably going to decrease with the work on [MEP-4](../MEP4/README.md) where there will be dedicated APIs for users and administrators of metal-stack including fine-grained access tokens. -## Implementation +With this MEP we want to describe a way to improve this current situation and allow other users that do not rely on the Gardener integration – for whatever motivation they have not to – to adequately manage firewalls. For this, we propose an alternative configuration for the firewall-controller that is more versatile and independent of the Gardener. -During reconciliation the CAPMS controller manager has a look at `MetalStackCluster.Spec.NodeNetworkID`. If it is not yet set, the node network should created and patched back into the spec. Then the manager should move to the `MetalStackCluster.Spec.FirewallTemplate`. When set, it should create or update the matching `FirewallDeployment` according to the nested data like the image, networks, size or the `FirewallDeployment.Spec.Template.InitialRuleSet`. -From here on the fcm will take over to reconcile all `FirewallDeployment`s. Currently it only watches a specific namespace and needs to be adapted to be capable of operating on all namespaces instead. Also initial firewall rule sets need to be implemented. +## Proposal -When the operator installs Cluster API including CAPMS into their cluster, `clusterctl init --infrastructure metal-stack` needs to include the fcm custom resources and optionally its controller deployment to be able to reconcile firewalls. +- The firewall rules of the firewall entity can be updated through the metal-api. +- The firewall-controller can be configured through a dedicated config file. +- Inside this config file the data source for all its dynamic configuration tasks can be set independently. +- For example the data source of the core firewall rules could be set the Gardener seed or the metal-api firewall entity, while the CWNPs should be fetched and applied from a given kubeconfig. +- This configuration file is intended to be injected through userdata along with potential source connection credentials. -When generating a new cluster using `clusterctl generate cluster --infrastructure metal-stack cluster-name`, this should also generate a firewall entry including some basic firewall rules. +```yaml +# the main configuration source contributes +# - firewall nftables rules +# - egress rules +# - prefixes +# - rate limiting +# - versions of components (firewall-controller, droptailer, ...) +main: + kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/seed.yaml + components: + - kind: Firewall + namespace: shoot-namespace + +# the additional configuration sources contributes +# - additional firewall nftables rules +additional: +- kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/shoot.yaml + components: + - kind: ClusterwideNetworkPolicy + namespace: firewall + - kind: Service + namespace: null + +- kind: metal-api + config: + url: https://metal-api + hmac: some-hmac + type: Metal-View + +- kind: static + config: + egress: [] + ingress: [] + +# the reports configuration output generates +# - FirewallMonitor +reports: +- kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/shoot.yaml + components: + - kind: FirewallMonitor + namespace: firewall + name: firewall-monitor # default name of firewall +``` + +### Non-Goals + +- Resolving the missing differentiation between users and administrators by lettings users pass userdata and SSH keys to the firewall creation. + - This is even more related to MEP-4 than this MEP. + +### Advantages + +- Offers a native metal-stack solution that improves managing firewalls for users by adding dynamic reconfiguration through the metal-api +- Improve consistency throughout the API (firewall rules would reflect what is in metal-api) +- Other providers like Cluster API can leverage this approach, too +- It can contribute to solving the shoot migration issue (in Cluster API case the `clusterctl move` for firewall objects) + - For Gardener takes the seed out of the equation (of which the kubeconfig changes during shoot migration) + - However: Things like egress rules, rate limiting, etc. are currently not part of the firewall entity in the metal-api (these would need to be added to the firewall entity as otherwise there is no feature parity) + +### Caveats + +- Metal-View access is too broad for firewalls. Mitigated by MEP-4. +- Polling of the firewall-controller is bad for performance. Mitigated by MEP-4. + +### Firewall Controller Manager + +- firewall-controller + - Allow OPTIONAL configuration source (metal-api with view HMAC), which is considered when no seed kubeconfig was passed in the userdata +- firewall-controller-manager + - Update the firewall entity in the metal-api +- metal-api + - Allow update of firewall rules in the machine allocation spec + - Add egress rules, rate limiting, etc to the firewall entity ```yaml -apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 -kind: MetalStackCluster -metadata: - name: cluster-name - namespace: default +kind: FirewallDeployment spec: - projectID: some-project-id - partition: partition - nodeNetworkID: node-network-id # now optional if firewall given - firewallTemplate: - size: machine-size - image: firewall-ubuntu - networks: [] # the node-network-id will be appended automatically - initialRuleSet: - ingress: - - comment: allow incoming https - protocol: tcp - ports: [443] - from: [0.0.0.0/0] - egress: - - comment: allow outgoing https - protocol: tcp - ports: [443] - to: [0.0.0.0/0] - # additional entries for dns, ssh and ntp required + userdataContents: + - path: /etc/firewall-controller/config.yaml + content: | + --- + main: + additional: + reports: + - path: /etc/firewall-controller/seed.yaml + secretRef: + name: seed-kubeconfig + - path: /etc/firewall-controller/shoot.yaml + secretRef: + name: shoot-kubeconfig + - path: /etc/firewall-controller/shoot.yaml + generateSecret: true # TODO which? :D ``` -Cluster API allows to move the resource and therefore the control over the bootstrapped workload cluster. As part of this process `clusterctl` marks the cluster as paused. Providers like CAPMS need to obey the pause and may not reconcile during this time period to avoid loss of infrastructure and data. +### Cluster API Provider Metal Stack -When using the fcm in this context, it should also obey this rule. But in order to keep the fcm separated from cluster api internals, we'd like to propose the `firewall.metal-stack.io/paused` annotation to pause the reconciliation. Managing this pause annotation on the `FirewallDeployment` is in the responsibility of the CAPMS controller manager. +The proposed changes would allow the cluster-api-provider-metal-stack to be able to manage firewalls to the same extent as metal-stack can with Gardener. +Here the firewall would need to be configured to use a kubeconfig to a cluster to reconcil`CWNP`s and the `FirewallMonitor`. This cluster might be a separate monitoring or configuration cluster that is accessible by the firewall and the FCM or the workload cluster itself if the CRDs are installed. -This annotation still needs to be propagated to nested resources like `FirewallSet` and `Firewall`. +Once the firewall-controller is able to update the status of the `FirewallMonitor`, the FCM is able to perform battle proof rolling updates of the firewall. +Due to the fact that the bootstrap cluster is typically being run in a kind cluster, there is no way for the firewall-controller to access the `Firewall` resource within. By using the metal-api as a data source, there is now a way for the firewall-controller to operate on. -Furthermore Cluster API also requires to have an unbroken chain of owner references. In practice this means `Firewall` is owned by `FirewallSet` which is in turn owned by `FirewallDeployment`. This is a [Kubernetes feature](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/) and good practice anyways. +- firewall-controller + - for FirewallMonitor and CWNP use user provided SecretRef to Kubeconfig + - for LoadBalancer use workload cluster kubeconfig + +```yaml +kind: MetalStackCluster +metadata: + # existing fields omitted +spec: + firewallTemplate: + # existing fields omitted + staticRuleSet: [] + controllerConfig: | + --- + main: + additional: + reports: + additionalFiles: +``` -## Affected Components +## Roadmap -For some working items, we already created proof of concept PRs, which might need to be split up and are not yet production ready. +In general this proposal is not thought to be implemented in one batch. Instead an incremental approach is required. -- Firewall Controller Manager - - must be configured to watch all namespaces for `FirewallDeployment` [fcm#66](https://github.com/metal-stack/firewall-controller-manager/pull/66) - - initial firewall rules needs to be implemented [fcm#64](https://github.com/metal-stack/firewall-controller-manager/pull/64) - - the annotations of the `FirewallDeployment` need to be propagated down to the `FirewallSet` and `Firewall` - - needs to introduce and implement the `firewall.metal-stack.io/paused` annotation - - set owner references to `FirewallSet` and `Firewall` -- Cluster API Controller Manager - - the `FirewallDeployment` pause annotation needs to be managed - - the `FirewallDeployment` resource should be created if template set [CAPMS#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) - - management of SSH keys for machines and firewalls [CAPMS#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) - - cluster-template - - needs to install CRDs for the FCM in order to be respected during moves [CAPMS#82](https://github.com/metal-stack/cluster-api-provider-metal-stack/pull/82) - - eventually also install the FCM itself - - `MetalStackCluster` resource definition - - add `MetalStackCluster.Spec.FirewallTemplate` - - make `Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given +1. Allow Cluster API to use the FCM (provide immutable firewalls that run without firewall-controller). + - Add `spec.staticRuleSet` to Firewall. + - Add `firewall.metal-stack.io/paused` annotation (managed by CAPMS during move, theoretically useful for Gardener shoot migration as well to avoid shallow deletion). + - Reconcile multiple `FirewallDeployment` resources per namespace. + - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance) +2. Extend firewall-controller with the configuration file (no different data sources than kubernetes) and let the FCM generate this configuration file along with the rest of the userdata. +3. Add metal-api as configuration source. + - Allow updates of firewall rules in the firewall entity. + - For Cluster API: Let the cluster controller generate the userdata including the configuration for this additional data source. +4. Move to more generic interface for the FCM (remove Gardener coupling) -## Caveats and Organizational Implications -When the cluster is pivoted and reconciles its own firewall, a malfunctioning firewall prevents the cluster from self-healing and requires manual intervention by creating a new firewall. This is an inherent problem of the cluster-api approach. It can be circumvented by using an extra cluster to manage workload clusters. +## Alternatives Considered -In the current form of this approach firewalls and therefore the firewall egress and ingress rules are managed by the cluster operators that manage the cluster-api resources. -Hence it will not be possible to gain a fine-grained control over every cluster operator's choices from a central ruleset at the level of metal-stack firewalls. -In case this control surfaces as a requirement, it would need to be implemented in a firewall external to metal-stack. +- instead of the generic data sources, for each mechanism a kubeconfig could be provided. Though this is not a scalable nor flexible approach. In the end the same internal data structure is still needed. From 0940cebc45e7e21c76145adba40d8c16ac30aa48 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Fri, 14 Mar 2025 16:25:21 +0100 Subject: [PATCH 07/18] docs: more details regarding cluster api --- .../src/development/proposals/MEP16/README.md | 57 ++++++++++++++----- 1 file changed, 43 insertions(+), 14 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index b81543c777..f8880a410b 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -12,15 +12,15 @@ For this reason, we have a gap in the metal-stack project in terms of a missing Parts of these problems are probably going to decrease with the work on [MEP-4](../MEP4/README.md) where there will be dedicated APIs for users and administrators of metal-stack including fine-grained access tokens. -With this MEP we want to describe a way to improve this current situation and allow other users that do not rely on the Gardener integration – for whatever motivation they have not to – to adequately manage firewalls. For this, we propose an alternative configuration for the firewall-controller that is more versatile and independent of the Gardener. - +With this MEP we want to describe a way to improve this current situation and allow other users that do not rely on the Gardener integration – for whatever motivation they have not to – to adequately manage firewalls. For this, we propose an alternative configuration for the firewall-controller that is native to metal-stack and more independent of the Gardener. ## Proposal +- The firewall-controller can use the metal-api as a configuration source. - The firewall rules of the firewall entity can be updated through the metal-api. - The firewall-controller can be configured through a dedicated config file. - Inside this config file the data source for all its dynamic configuration tasks can be set independently. -- For example the data source of the core firewall rules could be set the Gardener seed or the metal-api firewall entity, while the CWNPs should be fetched and applied from a given kubeconfig. +- For example the data source of the core firewall rules could be set the Gardener seed or the metal-api firewall entity, while the CWNPs should be fetched and applied from a given kubeconfig (the shoot Kubeconfig in the Gardener case). - This configuration file is intended to be injected through userdata along with potential source connection credentials. ```yaml @@ -81,8 +81,9 @@ reports: ### Advantages - Offers a native metal-stack solution that improves managing firewalls for users by adding dynamic reconfiguration through the metal-api -- Improve consistency throughout the API (firewall rules would reflect what is in metal-api) -- Other providers like Cluster API can leverage this approach, too + - e.g., in the mini-lab, users can now allocate a machine, then an IP address and announce this IP from the machine without having to re-create the firewall but by adding a firewall rule to the firewall entity. +- Improve consistency throughout the API (firewall rules would reflect what is persisted in metal-api). +- Other providers like Cluster API can leverage this approach, too. - It can contribute to solving the shoot migration issue (in Cluster API case the `clusterctl move` for firewall objects) - For Gardener takes the seed out of the equation (of which the kubeconfig changes during shoot migration) - However: Things like egress rules, rate limiting, etc. are currently not part of the firewall entity in the metal-api (these would need to be added to the firewall entity as otherwise there is no feature parity) @@ -124,32 +125,60 @@ spec: ### Cluster API Provider Metal Stack -The proposed changes would allow the cluster-api-provider-metal-stack to be able to manage firewalls to the same extent as metal-stack can with Gardener. -Here the firewall would need to be configured to use a kubeconfig to a cluster to reconcil`CWNP`s and the `FirewallMonitor`. This cluster might be a separate monitoring or configuration cluster that is accessible by the firewall and the FCM or the workload cluster itself if the CRDs are installed. +In Cluster API there are essentially two main clusters: the management cluster and the workload cluster. +Typically a local bootstrap cluster is created in kind which acts as the management cluster. It creates the workload cluster. Thereafter the ownership of the workload cluster is typically moved to a different cluster which will then become the management cluster. +The new management cluster might actually be the workload cluster itself. -Once the firewall-controller is able to update the status of the `FirewallMonitor`, the FCM is able to perform battle proof rolling updates of the firewall. -Due to the fact that the bootstrap cluster is typically being run in a kind cluster, there is no way for the firewall-controller to access the `Firewall` resource within. By using the metal-api as a data source, there is now a way for the firewall-controller to operate on. +In contrast to Gardener, Cluster API tries to be as non-opinionated and as standard as possible. It is common practice to not install any non-required components or CRDs into the workload cluster. Therefore we cannot expect custom resources like `ClusterwideNetworkPolicy` or `FirewallMonitor` to be installed in the workload cluster. Therefore it's the responsibility of the operator to tell cluster-api-provider-metal-stack the kubeconfig for the cluster where these CRDs are installed and defined in. -- firewall-controller - - for FirewallMonitor and CWNP use user provided SecretRef to Kubeconfig - - for LoadBalancer use workload cluster kubeconfig +A viable configuration for a `MetalStackCluster` that generates firewall rules based of `Service` type `LoadBalancer` and `ClusterwideNetworkPolicy` and expects them to be deployed in the workload cluster is shown below. The `FirewallMonitor` will be reported into the same cluster. ```yaml kind: MetalStackCluster metadata: - # existing fields omitted + name: ${CLUSTER_NAME} spec: firewallTemplate: # existing fields omitted staticRuleSet: [] + additionalFiles: + - path: /etc/firewall-controller/workload.yaml + secretRef: + # this is the kubeconfig generated by kubeadm + name: ${CLUSTER_NAME}-kubeconfig + controllerConfig: | --- main: + kind: metal-api + config: + url: ${METAL_API_URL} + hmac: ${METAL_API_HMAC} + type: ${METAL_API_HMAC_TYPE} additional: + - kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/workload.yaml + components: + - kind: ClusterwideNetworkPolicy + namespace: firewall + - kind: Service + reports: - additionalFiles: + - kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/workload.yaml + components: + - kind: FirewallMonitor + namespace: firewall + name: firewall-monitor-${CLUSTER_NAME} + ``` +This approach allows maximum flexibility as intended by Cluster API and is still able to provide robust rolling updates of firewalls. + +An advanced use case of this flexibility would be a management cluster, that is in charge of multiple workload clusters. Where one workload cluster acts as a monitoring or tooling cluster, receives logs and the firewall monitor for the other workload clusters. The CWNPs could be defined here, all in a separate namespace. + ## Roadmap In general this proposal is not thought to be implemented in one batch. Instead an incremental approach is required. From b1b8c0c31b4ce34a80db89c9b33322b22d0d14fd Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Mon, 17 Mar 2025 11:24:57 +0100 Subject: [PATCH 08/18] docs: update with new chart --- .../src/development/proposals/MEP16/README.md | 51 +++++++++++-------- .../MEP16/firewall-for-capms-overview.svg | 2 +- 2 files changed, 30 insertions(+), 23 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index f8880a410b..6ddc02bb7a 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -120,11 +120,13 @@ spec: secretRef: name: shoot-kubeconfig - path: /etc/firewall-controller/shoot.yaml - generateSecret: true # TODO which? :D + generateGardenerFirewallControllerSecret: true ``` ### Cluster API Provider Metal Stack +![architectural overview](firewall-for-capms-overview.svg) + In Cluster API there are essentially two main clusters: the management cluster and the workload cluster. Typically a local bootstrap cluster is created in kind which acts as the management cluster. It creates the workload cluster. Thereafter the ownership of the workload cluster is typically moved to a different cluster which will then become the management cluster. The new management cluster might actually be the workload cluster itself. @@ -147,32 +149,37 @@ spec: # this is the kubeconfig generated by kubeadm name: ${CLUSTER_NAME}-kubeconfig - controllerConfig: | - --- - main: - kind: metal-api - config: - url: ${METAL_API_URL} - hmac: ${METAL_API_HMAC} - type: ${METAL_API_HMAC_TYPE} - additional: - - kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/workload.yaml - components: - - kind: ClusterwideNetworkPolicy - namespace: firewall - - kind: Service + controllerConfigSecretRef: + secretName: ${CLUSTER_NAME}-firewall-controller-config +--- +kind: Secret +metadata: + name: ${CLUSTER_NAME}-firewall-controller-config +stringData: + controllerConfig: | + --- + main: + config: + url: ${METAL_API_URL} + hmac: ${METAL_API_HMAC} + type: ${METAL_API_HMAC_TYPE} + additional: + - kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/workload.yaml + components: + - kind: ClusterwideNetworkPolicy + namespace: firewall + - kind: Service - reports: - - kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/workload.yaml + reports: + - kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/workload.yaml components: - kind: FirewallMonitor namespace: firewall name: firewall-monitor-${CLUSTER_NAME} - ``` This approach allows maximum flexibility as intended by Cluster API and is still able to provide robust rolling updates of firewalls. diff --git a/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg index 07fe7999b4..88b6019567 100644 --- a/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg +++ b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg @@ -1,4 +1,4 @@ -
Handles traffic
Handles traffic
Firewall
Firewall
Firewall Controller
Firewall Controller
Workload Cluster
Workload Cluster
Configures
Configures
Management Cluster
Management Cluster
Cluster API Provider metal-stack
Cluster API Provider...
MetalStackCluster
MetalStackCluster
Firewall Deployment
Firewall Deployment
Firewall CRD
Firewall CRD
Firewall Controller Manager
Firewall Controller...
Pushes Configuration
Pushes Configuration
Configures
Configures
Operator
Opera... \ No newline at end of file +
handles traffic
Firewall
Firewall Controller
node-exporter
nftables-exporter
droptailer-server
Workload Cluster
droptailer-client
Configures
Bootstrap or Management Cluster
reconcile
configures
reconcile
Cluster API Provider metal-stack
Metal Stack Cluster CRD
Firewall Deployment CRD
Firewall CRD
Firewall Set CRD
rec
reconcile
reconcile
Firewall Controller Manager
Metal Stack Machine CRD
manages
Operator
Kubeconfig FirewallMonitor
FirewallMonitor CRD
main metal-api
Firewall entity
kubeconfig CWNP
Clusterwide Network Policy CRD
base config
controllerConfig
user-defined
network rules
reports firewall
state
collects dropped packages
controllerConfig
controllerConfig
 \ No newline at end of file From e21cb2c10b1c8cd8f47bc6699a3dd3c5ebbe13c0 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Mon, 17 Mar 2025 15:38:27 +0100 Subject: [PATCH 09/18] docs: wording --- .../src/development/proposals/MEP16/README.md | 24 +++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 6ddc02bb7a..5bd86a4dfd 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -95,13 +95,13 @@ reports: ### Firewall Controller Manager -- firewall-controller - - Allow OPTIONAL configuration source (metal-api with view HMAC), which is considered when no seed kubeconfig was passed in the userdata -- firewall-controller-manager - - Update the firewall entity in the metal-api -- metal-api - - Allow update of firewall rules in the machine allocation spec - - Add egress rules, rate limiting, etc to the firewall entity +The firewall-controller should allow retrieving the main configuration of the firewall from the metal-api (with view HMAC) as an alternative to the kubeconfig based approach that is used in our Gardener setup. + +In this scenario, the firewall-controller-manager is responsible to update the firewall entity in the metal-api whenever the firewall rules change of the firewall resource. + +Of course the metail-api needs to update firewall rules in the machine allocation spec. Additionally ingress and egress rules need to be added to the firewall entity. + +The desired behavior will be configured in the `FirewallDeployment`. Specifically for the Gardener use case, the generation of the shoot kubeconfig for the firewall to be able to access the `Firewall` resource definition in the seed cluster will be hidden behind the `generateGardenerFirewallControllerSecret` flag. ```yaml kind: FirewallDeployment @@ -186,6 +186,14 @@ This approach allows maximum flexibility as intended by Cluster API and is still An advanced use case of this flexibility would be a management cluster, that is in charge of multiple workload clusters. Where one workload cluster acts as a monitoring or tooling cluster, receives logs and the firewall monitor for the other workload clusters. The CWNPs could be defined here, all in a separate namespace. +#### Cluster API Caveats + +When the cluster is pivoted and reconciles its own firewall, a malfunctioning firewall prevents the cluster from self-healing and requires manual intervention by creating a new firewall. This is an inherent problem of the cluster-api approach. It can be circumvented by using an extra cluster to manage workload clusters. + +In the current form of this approach firewalls and therefore the firewall egress and ingress rules are managed by the cluster operators that manage the cluster-api resources. +Hence it will not be possible to gain a fine-grained control over every cluster operator's choices from a central ruleset at the level of metal-stack firewalls. +In case this control surfaces as a requirement, it would need to be implemented in a firewall external to metal-stack. + ## Roadmap In general this proposal is not thought to be implemented in one batch. Instead an incremental approach is required. @@ -195,6 +203,8 @@ In general this proposal is not thought to be implemented in one batch. Instead - Add `firewall.metal-stack.io/paused` annotation (managed by CAPMS during move, theoretically useful for Gardener shoot migration as well to avoid shallow deletion). - Reconcile multiple `FirewallDeployment` resources per namespace. - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance) + - add `MetalStackCluster.Spec.FirewallTemplate` + - make `MetalStackCluster.Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given 2. Extend firewall-controller with the configuration file (no different data sources than kubernetes) and let the FCM generate this configuration file along with the rest of the userdata. 3. Add metal-api as configuration source. - Allow updates of firewall rules in the firewall entity. From 66c2f485d506b601439f4782077c3ebb2e5e7ba5 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Tue, 18 Mar 2025 13:34:45 +0100 Subject: [PATCH 10/18] docs: links --- docs/src/development/proposals/MEP16/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 5bd86a4dfd..189a14bef6 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -76,7 +76,7 @@ reports: ### Non-Goals - Resolving the missing differentiation between users and administrators by lettings users pass userdata and SSH keys to the firewall creation. - - This is even more related to MEP-4 than this MEP. + - This is even more related to [MEP-4](../MEP4/README.md) than this MEP. ### Advantages @@ -90,8 +90,8 @@ reports: ### Caveats -- Metal-View access is too broad for firewalls. Mitigated by MEP-4. -- Polling of the firewall-controller is bad for performance. Mitigated by MEP-4. +- Metal-View access is too broad for firewalls. Mitigated by [MEP-4](../MEP4/README.md). +- Polling of the firewall-controller is bad for performance. Mitigated by [MEP-4](../MEP4/README.md). ### Firewall Controller Manager From 6aa7290246d6d435f4ed8410ed445f7c33a083ba Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Wed, 19 Mar 2025 09:16:26 +0100 Subject: [PATCH 11/18] docs: kubeconfig secret --- docs/src/development/proposals/MEP16/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 189a14bef6..556f4ae2f9 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -1,10 +1,10 @@ # metal-api as an Alternative Configuration Source for the firewall-controller -In the current situation, a firewall as provisioned by metal-stack is a fully immutable entity. Any modifications on the firewall like changing the firewall ruleset must be done _somehow_ by the user – the metal-api and hence the metal-stack is not aware of its current state. +In the current situation, a firewall as provisioned by metal-stack is a fully immutable entity. Any modifications on the firewall like changing the firewall ruleset must be done _somehow_ by the user – the metal-api and hence metal-stack is not aware of its current state. As part of our [integration with the Gardener project](https://docs.metal-stack.io/stable/overview/kubernetes/#Gardener) we offer a solution called the [firewall-controller](https://github.com/metal-stack/firewall-controller), which is part of our [firewall OS images](https://github.com/metal-stack/metal-images/blob/6318a624861b18a559a9d37299bca5f760eef524/firewall/Dockerfile#L57-L58) and addresses shortcomings of the firewall resource's immutability, which would otherwise be completely impractible to work with. The firewall-controller crashes infinitely if it is not properly configured through the userdata when using the firewall image of metal-stack. -The firewall-controller approach is tightly coupled to the Gardener and it requires the administrator of the Gardener installation to pass a shoot and a seed kubeconfig through machine userdata when creating the firewall. How this userdata has to look like is not documented and is just part of another project called the [firewall-controller-manager](https://github.com/metal-stack/firewall-controller-manager), which task is to orchestrate rolling updates of firewall machines in a way that network traffic interruption is minimal when updating a firewall or applying a change to an immutable firewall configuration. +The firewall-controller approach is tightly coupled to Gardener and it requires the administrator of the Gardener installation to pass a shoot and a seed kubeconfig through machine userdata when creating the firewall. How this userdata has to look like is not documented and is just part of another project called the [firewall-controller-manager](https://github.com/metal-stack/firewall-controller-manager), which task is to orchestrate rolling updates of firewall machines in a way that network traffic interruption is minimal when updating a firewall or applying a change to an immutable firewall configuration. In general, a firewall entity in metal-stack has similarities to the machine entity but it has a fundamental difference: A user gains ownership over a machine after provisioning. They can access it through SSH, modify it at will and this is completely wanted. For firewalls, however, we do not want a user to access the provisioned firewall as the firewall is a privileged part of the infrastructure with access to the underlay network. The underlay can not be tampered with at any given point in time by a user as it can destroy the entire network traffic flow inside a metal-stack partition. @@ -182,6 +182,8 @@ stringData: name: firewall-monitor-${CLUSTER_NAME} ``` +Here the firewall-controller-config will be referenced by the `MetalStackCluster` as a `Secret`. Please note that the `secretRef`s in `additionalFiles` will not be fetched and will directly be passed to the `FirewallDeployment`. At first the reconciliation of it in the FCM will fail due to the missing Kubeconfig secret. After the `MetalStackCluster` has been marked as ready, CAPI will create this missing secret. Effectively the firewall and initial control plane node should be created at the same time. + This approach allows maximum flexibility as intended by Cluster API and is still able to provide robust rolling updates of firewalls. An advanced use case of this flexibility would be a management cluster, that is in charge of multiple workload clusters. Where one workload cluster acts as a monitoring or tooling cluster, receives logs and the firewall monitor for the other workload clusters. The CWNPs could be defined here, all in a separate namespace. From 02c77f133775888fb4ea60c387e8cf40311f2960 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Thu, 20 Mar 2025 12:58:57 +0100 Subject: [PATCH 12/18] docs: update --- docs/src/development/proposals/MEP16/README.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 556f4ae2f9..9ea610ac43 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -12,16 +12,18 @@ For this reason, we have a gap in the metal-stack project in terms of a missing Parts of these problems are probably going to decrease with the work on [MEP-4](../MEP4/README.md) where there will be dedicated APIs for users and administrators of metal-stack including fine-grained access tokens. -With this MEP we want to describe a way to improve this current situation and allow other users that do not rely on the Gardener integration – for whatever motivation they have not to – to adequately manage firewalls. For this, we propose an alternative configuration for the firewall-controller that is native to metal-stack and more independent of the Gardener. +With this MEP we want to describe a way to improve this current situation and allow other users that do not rely on the Gardener integration – for whatever motivation they have not to – to adequately manage firewalls. For this, we propose an alternative configuration for the firewall-controller that is native to metal-stack and more independent of Gardener. ## Proposal -- The firewall-controller can use the metal-api as a configuration source. -- The firewall rules of the firewall entity can be updated through the metal-api. -- The firewall-controller can be configured through a dedicated config file. -- Inside this config file the data source for all its dynamic configuration tasks can be set independently. -- For example the data source of the core firewall rules could be set the Gardener seed or the metal-api firewall entity, while the CWNPs should be fetched and applied from a given kubeconfig (the shoot Kubeconfig in the Gardener case). -- This configuration file is intended to be injected through userdata along with potential source connection credentials. +The central idea of this proposal is allowing the firewall-controller to use the metal-api as a configuration source. This should serve as an alternative strategy to the currently used Seed Kubeconfig based approach in the Gardener use-case. +Updates of the firewall rules of the firewall entity should be possible through the metal-api. + +The firewall-controller itself should now be able to decide which of the two main strategies should be used for the base configuration: a kubeconfig or the metal-api. This should be possible through a dedicated _firewall-controller-config_. + +Using this config will now allow operators to fine-tune the data sources for all of its dynamic configuration tasks independently. +For example the data source of the core firewall rules could be set the Gardener seed or the metal-api firewall entity, while the CWNPs should be fetched and applied from a given kubeconfig (the shoot Kubeconfig in the Gardener case). +This configuration file is intended to be injected through userdata along with potential source connection credentials. ```yaml # the main configuration source contributes From 08f22dc938f058f53bb4d5d94022f1c8a3f66d54 Mon Sep 17 00:00:00 2001 From: Gerrit Date: Thu, 20 Mar 2025 14:54:57 +0100 Subject: [PATCH 13/18] Review. --- .../src/development/proposals/MEP16/README.md | 101 +++++++++++++----- 1 file changed, 75 insertions(+), 26 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 9ea610ac43..e423112e4c 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -1,14 +1,14 @@ # metal-api as an Alternative Configuration Source for the firewall-controller -In the current situation, a firewall as provisioned by metal-stack is a fully immutable entity. Any modifications on the firewall like changing the firewall ruleset must be done _somehow_ by the user – the metal-api and hence metal-stack is not aware of its current state. +In the current situation, a firewall as provisioned by metal-stack is a fully immutable entity. Any modifications on the firewall like changing the firewall ruleset must be done _somehow_ by the user – the metal-api and hence metal-stack is not aware of its current state. As part of our [integration with the Gardener project](https://docs.metal-stack.io/stable/overview/kubernetes/#Gardener) we offer a solution called the [firewall-controller](https://github.com/metal-stack/firewall-controller), which is part of our [firewall OS images](https://github.com/metal-stack/metal-images/blob/6318a624861b18a559a9d37299bca5f760eef524/firewall/Dockerfile#L57-L58) and addresses shortcomings of the firewall resource's immutability, which would otherwise be completely impractible to work with. The firewall-controller crashes infinitely if it is not properly configured through the userdata when using the firewall image of metal-stack. -The firewall-controller approach is tightly coupled to Gardener and it requires the administrator of the Gardener installation to pass a shoot and a seed kubeconfig through machine userdata when creating the firewall. How this userdata has to look like is not documented and is just part of another project called the [firewall-controller-manager](https://github.com/metal-stack/firewall-controller-manager), which task is to orchestrate rolling updates of firewall machines in a way that network traffic interruption is minimal when updating a firewall or applying a change to an immutable firewall configuration. +The firewall-controller approach is tightly coupled to Gardener and it requires the administrator of the Gardener installation to pass a shoot and a seed kubeconfig through machine userdata when creating the firewall. How this userdata has to look like is not documented and is just part of another project called the [firewall-controller-manager](https://github.com/metal-stack/firewall-controller-manager) (FCM), which task is to orchestrate rolling updates of firewall machines in a way that network traffic interruption is minimal when updating a firewall or applying a change to an immutable firewall configuration. In general, a firewall entity in metal-stack has similarities to the machine entity but it has a fundamental difference: A user gains ownership over a machine after provisioning. They can access it through SSH, modify it at will and this is completely wanted. For firewalls, however, we do not want a user to access the provisioned firewall as the firewall is a privileged part of the infrastructure with access to the underlay network. The underlay can not be tampered with at any given point in time by a user as it can destroy the entire network traffic flow inside a metal-stack partition. -For this reason, we have a gap in the metal-stack project in terms of a missing solution for people who do not rely on the Gardener integration. We are basically leaving a user with the option to implement an orchestrated recreation of every possible change on the firewall to minimize traffic interruption for the machines sitting behind the firewall or re-implement the firewall-controller to how they want to use it for their use-case. Also we do not have a clear distinction in the API between user and landscape operator for firewalls. If a user would allocate firewall it is also possible for the user to inject his own SSH keys and access the firewall and tamper with the underlay network. +For this reason, we have a gap in the metal-stack project in terms of a missing solution for people who do not rely on the Gardener integration. We are basically leaving a user with the option to implement an orchestrated recreation of every possible change on the firewall to minimize traffic interruption for the machines sitting behind the firewall or re-implement the firewall-controller to how they want to use it for their use-case. Also we do not have a clear distinction in the API between user and landscape operator for firewalls. If a user would allocate firewall it is also possible for the user to inject his own SSH keys and access the firewall and tamper with the underlay network. Parts of these problems are probably going to decrease with the work on [MEP-4](../MEP4/README.md) where there will be dedicated APIs for users and administrators of metal-stack including fine-grained access tokens. @@ -16,14 +16,14 @@ With this MEP we want to describe a way to improve this current situation and al ## Proposal -The central idea of this proposal is allowing the firewall-controller to use the metal-api as a configuration source. This should serve as an alternative strategy to the currently used Seed Kubeconfig based approach in the Gardener use-case. +The central idea of this proposal is allowing the firewall-controller to use the metal-api as a configuration source. This should serve as an alternative strategy to the currently used FCM `Firewall` resource based approach in the Gardener use-case. Updates of the firewall rules of the firewall entity should be possible through the metal-api. The firewall-controller itself should now be able to decide which of the two main strategies should be used for the base configuration: a kubeconfig or the metal-api. This should be possible through a dedicated _firewall-controller-config_. Using this config will now allow operators to fine-tune the data sources for all of its dynamic configuration tasks independently. -For example the data source of the core firewall rules could be set the Gardener seed or the metal-api firewall entity, while the CWNPs should be fetched and applied from a given kubeconfig (the shoot Kubeconfig in the Gardener case). -This configuration file is intended to be injected through userdata along with potential source connection credentials. +For example the data source of the core firewall rules could be set either from the `Firewall` resource located in the Gardener `Seed` or the metal-api firewall entity, while the CWNPs should be fetched and applied from a given kubeconfig (the `Shoot` Kubeconfig in the Gardener case). +This configuration file is intended to be injected during firewall creation through the userdata along with potential source connection credentials. ```yaml # the main configuration source contributes @@ -39,6 +39,15 @@ main: components: - kind: Firewall namespace: shoot-namespace + enableKubeconfigRotation: /etc/firewall-controller/shoot.yaml + + # alternatively users can use the metal-api configuration source + # for now, the metal-api configuration does not contribute egress rules or rate limiting # + # kind: metal-api + # config: + # url: https://metal-api + # hmac: some-hmac + # type: Metal-View # the additional configuration sources contributes # - additional firewall nftables rules @@ -75,6 +84,48 @@ reports: name: firewall-monitor # default name of firewall ``` +For example, in order to maintain the existing Gardener integration, the configuration file for the firewall-controller will look like this: + +```yaml +main: + kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/seed.yaml + components: + - kind: Firewall + namespace: shoot-namespace + +additional: +- kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/shoot.yaml + components: + - kind: ClusterwideNetworkPolicy + namespace: firewall + - kind: Service + namespace: null + +reports: +- kind: kubernetes + config: + kubeconfigPath: /etc/firewall-controller/shoot.yaml + components: + - kind: FirewallMonitor + namespace: firewall + name: firewall-monitor # default name of firewall +``` + +Plain metal-stack users might use a configuration like this: + +```yaml +main: + kind: metal-api + config: + url: https://metal-api + hmac: some-hmac + type: Metal-View +``` + ### Non-Goals - Resolving the missing differentiation between users and administrators by lettings users pass userdata and SSH keys to the firewall creation. @@ -97,13 +148,13 @@ reports: ### Firewall Controller Manager -The firewall-controller should allow retrieving the main configuration of the firewall from the metal-api (with view HMAC) as an alternative to the kubeconfig based approach that is used in our Gardener setup. +As described, the firewall-controller should allow retrieving the main configuration of the firewall from the metal-api (with view HMAC) as an alternative to the kubeconfig based approach that is used in our Gardener setup. In this scenario, the firewall-controller-manager is responsible to update the firewall entity in the metal-api whenever the firewall rules change of the firewall resource. -Of course the metail-api needs to update firewall rules in the machine allocation spec. Additionally ingress and egress rules need to be added to the firewall entity. +Of course the metal-api needs to update firewall rules in the machine allocation spec. Additionally ingress and egress rules need to be added to the firewall entity. -The desired behavior will be configured in the `FirewallDeployment`. Specifically for the Gardener use case, the generation of the shoot kubeconfig for the firewall to be able to access the `Firewall` resource definition in the seed cluster will be hidden behind the `generateGardenerFirewallControllerSecret` flag. +The desired behavior will be configured in the `FirewallDeployment`. Specifically for the Gardener use case, the generation of the kubeconfig for the firewall-controller to be able to access the `Firewall` resource in the seed cluster will be hidden behind the `generateFirewallControllerKubeconfig` flag. ```yaml kind: FirewallDeployment @@ -118,11 +169,10 @@ spec: - path: /etc/firewall-controller/seed.yaml secretRef: name: seed-kubeconfig + generateFirewallControllerKubeconfig: true - path: /etc/firewall-controller/shoot.yaml secretRef: name: shoot-kubeconfig - - path: /etc/firewall-controller/shoot.yaml - generateGardenerFirewallControllerSecret: true ``` ### Cluster API Provider Metal Stack @@ -130,10 +180,10 @@ spec: ![architectural overview](firewall-for-capms-overview.svg) In Cluster API there are essentially two main clusters: the management cluster and the workload cluster. -Typically a local bootstrap cluster is created in kind which acts as the management cluster. It creates the workload cluster. Thereafter the ownership of the workload cluster is typically moved to a different cluster which will then become the management cluster. +Typically a local bootstrap cluster is created in kind which acts as the management cluster. It creates the workload cluster. Thereafter the ownership of the workload cluster is typically moved (using `clusterctl move`) to a different cluster which will then become the management cluster. The new management cluster might actually be the workload cluster itself. -In contrast to Gardener, Cluster API tries to be as non-opinionated and as standard as possible. It is common practice to not install any non-required components or CRDs into the workload cluster. Therefore we cannot expect custom resources like `ClusterwideNetworkPolicy` or `FirewallMonitor` to be installed in the workload cluster. Therefore it's the responsibility of the operator to tell cluster-api-provider-metal-stack the kubeconfig for the cluster where these CRDs are installed and defined in. +In contrast to Gardener, Cluster API aims to be less opinionated. It is common practice to not install any non-required components or CRDs into the workload cluster by default. Therefore we cannot expect custom resources like `ClusterwideNetworkPolicy` or `FirewallMonitor` to be installed in the workload cluster. Therefore it's the responsibility of the operator to tell [cluster-api-provider-metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack) the kubeconfig for the cluster where these CRDs are installed and defined in. A viable configuration for a `MetalStackCluster` that generates firewall rules based of `Service` type `LoadBalancer` and `ClusterwideNetworkPolicy` and expects them to be deployed in the workload cluster is shown below. The `FirewallMonitor` will be reported into the same cluster. @@ -145,14 +195,14 @@ spec: firewallTemplate: # existing fields omitted staticRuleSet: [] - additionalFiles: - - path: /etc/firewall-controller/workload.yaml - secretRef: - # this is the kubeconfig generated by kubeadm - name: ${CLUSTER_NAME}-kubeconfig - - controllerConfigSecretRef: + + userdataContents: + - path: /etc/firewall-controller/config.yaml secretName: ${CLUSTER_NAME}-firewall-controller-config + + - path: /etc/firewall-controller/workload.yaml + # this is the kubeconfig generated by kubeadm + secretName: ${CLUSTER_NAME}-kubeconfig --- kind: Secret metadata: @@ -173,7 +223,7 @@ stringData: - kind: ClusterwideNetworkPolicy namespace: firewall - kind: Service - + reports: - kind: kubernetes config: @@ -184,7 +234,7 @@ stringData: name: firewall-monitor-${CLUSTER_NAME} ``` -Here the firewall-controller-config will be referenced by the `MetalStackCluster` as a `Secret`. Please note that the `secretRef`s in `additionalFiles` will not be fetched and will directly be passed to the `FirewallDeployment`. At first the reconciliation of it in the FCM will fail due to the missing Kubeconfig secret. After the `MetalStackCluster` has been marked as ready, CAPI will create this missing secret. Effectively the firewall and initial control plane node should be created at the same time. +Here the firewall-controller-config will be referenced by the `MetalStackCluster` as a `Secret`. Please note that the `Secret`s in `userdataContents` will not be fetched and will directly be passed to the `FirewallDeployment`. At first the reconciliation of it in the FCM will fail due to the missing Kubeconfig secret. After the `MetalStackCluster` has been marked as ready, CAPI will create this missing secret. Effectively the firewall and initial control plane node should be created at the same time. This approach allows maximum flexibility as intended by Cluster API and is still able to provide robust rolling updates of firewalls. @@ -206,16 +256,15 @@ In general this proposal is not thought to be implemented in one batch. Instead - Add `spec.staticRuleSet` to Firewall. - Add `firewall.metal-stack.io/paused` annotation (managed by CAPMS during move, theoretically useful for Gardener shoot migration as well to avoid shallow deletion). - Reconcile multiple `FirewallDeployment` resources per namespace. - - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance) - - add `MetalStackCluster.Spec.FirewallTemplate` - - make `MetalStackCluster.Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given + - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance). + - Add `MetalStackCluster.Spec.FirewallTemplate`. + - Make `MetalStackCluster.Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given. 2. Extend firewall-controller with the configuration file (no different data sources than kubernetes) and let the FCM generate this configuration file along with the rest of the userdata. 3. Add metal-api as configuration source. - Allow updates of firewall rules in the firewall entity. - For Cluster API: Let the cluster controller generate the userdata including the configuration for this additional data source. 4. Move to more generic interface for the FCM (remove Gardener coupling) - ## Alternatives Considered - instead of the generic data sources, for each mechanism a kubeconfig could be provided. Though this is not a scalable nor flexible approach. In the end the same internal data structure is still needed. From 51eb27ae89530578b3cd59e1369cea2243783d55 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Thu, 3 Apr 2025 13:25:40 +0200 Subject: [PATCH 14/18] refactor: simpler controller config --- .../src/development/proposals/MEP16/README.md | 256 ++++++++++-------- 1 file changed, 141 insertions(+), 115 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index e423112e4c..c0b0c13495 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -1,5 +1,49 @@ # metal-api as an Alternative Configuration Source for the firewall-controller +TODO: +- list of controllers in firewall controllers + - cwnp + - droptailer + - firewall monitor + - service type load balancer + - firewall entity + - firewall resource + - self update +- attach to node network +- more specific instead of generic config + +```yaml +name: best-firewall-ever + +controllers: + firewall: + # one of kubernetes, metalAPI or static + kubernetes: + kubeconfig: /path/to/seed.yaml + metalAPI: + url: https://metal-api + hmac: some-hmac + type: Metal-View + projectID: abc + static: + egress: [] + ingress: [] + + selfUpdate: + enabled: true + droptailer: + enabled: true + + service: + kubeconfig: /path/to/shoot.yaml + cwnp: + kubeconfig: /path/to/shoot.yaml + namespace: firewall + monitor: + kubeconfig: /path/to/shoot.yaml + namespace: firewall +``` + In the current situation, a firewall as provisioned by metal-stack is a fully immutable entity. Any modifications on the firewall like changing the firewall ruleset must be done _somehow_ by the user – the metal-api and hence metal-stack is not aware of its current state. As part of our [integration with the Gardener project](https://docs.metal-stack.io/stable/overview/kubernetes/#Gardener) we offer a solution called the [firewall-controller](https://github.com/metal-stack/firewall-controller), which is part of our [firewall OS images](https://github.com/metal-stack/metal-images/blob/6318a624861b18a559a9d37299bca5f760eef524/firewall/Dockerfile#L57-L58) and addresses shortcomings of the firewall resource's immutability, which would otherwise be completely impractible to work with. The firewall-controller crashes infinitely if it is not properly configured through the userdata when using the firewall image of metal-stack. @@ -8,11 +52,13 @@ The firewall-controller approach is tightly coupled to Gardener and it requires In general, a firewall entity in metal-stack has similarities to the machine entity but it has a fundamental difference: A user gains ownership over a machine after provisioning. They can access it through SSH, modify it at will and this is completely wanted. For firewalls, however, we do not want a user to access the provisioned firewall as the firewall is a privileged part of the infrastructure with access to the underlay network. The underlay can not be tampered with at any given point in time by a user as it can destroy the entire network traffic flow inside a metal-stack partition. -For this reason, we have a gap in the metal-stack project in terms of a missing solution for people who do not rely on the Gardener integration. We are basically leaving a user with the option to implement an orchestrated recreation of every possible change on the firewall to minimize traffic interruption for the machines sitting behind the firewall or re-implement the firewall-controller to how they want to use it for their use-case. Also we do not have a clear distinction in the API between user and landscape operator for firewalls. If a user would allocate firewall it is also possible for the user to inject his own SSH keys and access the firewall and tamper with the underlay network. +For this reason, we have a gap in the metal-stack project in terms of a missing solution for people who do not rely on the Gardener integration. We are basically leaving a user with the option to implement an orchestrated recreation of every possible change on the firewall to minimize traffic interruption for the machines sitting behind the firewall or re-implement the firewall-controller to how they want to use it for their use-case. + +Also we do not have a clear distinction in the API between user and metal-stack operator for firewalls. If a user would allocate a firewall it is also possible for the user to inject his own SSH keys and access the firewall and tamper with the underlay network. Parts of these problems are probably going to decrease with the work on [MEP-4](../MEP4/README.md) where there will be dedicated APIs for users and administrators of metal-stack including fine-grained access tokens. -With this MEP we want to describe a way to improve this current situation and allow other users that do not rely on the Gardener integration – for whatever motivation they have not to – to adequately manage firewalls. For this, we propose an alternative configuration for the firewall-controller that is native to metal-stack and more independent of Gardener. +With this MEP we want to describe a way to improve this current situation and allow other users that do not rely on the Gardener integration – for whatever motivation they have – to adequately manage firewalls. For this, we propose an alternative configuration for the firewall-controller that is native to metal-stack and more independent of Gardener. ## Proposal @@ -26,104 +72,84 @@ For example the data source of the core firewall rules could be set either from This configuration file is intended to be injected during firewall creation through the userdata along with potential source connection credentials. ```yaml -# the main configuration source contributes -# - firewall nftables rules -# - egress rules -# - prefixes -# - rate limiting -# - versions of components (firewall-controller, droptailer, ...) -main: - kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/seed.yaml - components: - - kind: Firewall - namespace: shoot-namespace - enableKubeconfigRotation: /etc/firewall-controller/shoot.yaml - - # alternatively users can use the metal-api configuration source - # for now, the metal-api configuration does not contribute egress rules or rate limiting # - # kind: metal-api - # config: - # url: https://metal-api - # hmac: some-hmac - # type: Metal-View - -# the additional configuration sources contributes -# - additional firewall nftables rules -additional: -- kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/shoot.yaml - components: - - kind: ClusterwideNetworkPolicy - namespace: firewall - - kind: Service - namespace: null - -- kind: metal-api - config: - url: https://metal-api - hmac: some-hmac - type: Metal-View - -- kind: static - config: - egress: [] - ingress: [] - -# the reports configuration output generates -# - FirewallMonitor -reports: -- kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/shoot.yaml - components: - - kind: FirewallMonitor - namespace: firewall - name: firewall-monitor # default name of firewall +# the name of the firewall, defaulted to the hostname +name: best-firewall-ever + +# all sub-controllers running on the firewall +# each can be configured independently +controllers: + # this is the base controller + firewall: + + # one of kubernetes, metalAPI or static + kubernetes: + kubeconfig: /path/to/seed.yaml # current gardener behavior + metalAPI: + url: https://metal-api + hmac: some-hmac + type: Metal-View + projectID: abc + static: + egress: [] + ingress: [] + + # these are optional: when not provided, they are disabled + selfUpdate: + enabled: true + droptailer: + enabled: true + + # these are optional: when not provided, they are disabled + service: + kubeconfig: /path/to/shoot.yaml + cwnp: + kubeconfig: /path/to/shoot.yaml + namespace: firewall + monitor: + kubeconfig: /path/to/shoot.yaml + namespace: firewall ``` For example, in order to maintain the existing Gardener integration, the configuration file for the firewall-controller will look like this: ```yaml -main: - kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/seed.yaml - components: - - kind: Firewall - namespace: shoot-namespace - -additional: -- kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/shoot.yaml - components: - - kind: ClusterwideNetworkPolicy - namespace: firewall - - kind: Service - namespace: null - -reports: -- kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/shoot.yaml - components: - - kind: FirewallMonitor - namespace: firewall - name: firewall-monitor # default name of firewall +name: shoot--abc--cluster-firewall-def + +controllers: + firewall: + + kubernetes: + kubeconfig: /etc/firewall-controller/seed.yaml + + selfUpdate: + enabled: true + droptailer: + enabled: true + + service: + kubeconfig: /etc/firewall-controller/shoot.yaml + cwnp: + kubeconfig: /etc/firewall-controller/shoot.yaml + namespace: firewall + monitor: + kubeconfig: /etc/firewall-controller/shoot.yaml + namespace: firewall ``` Plain metal-stack users might use a configuration like this: ```yaml -main: - kind: metal-api - config: - url: https://metal-api - hmac: some-hmac - type: Metal-View +name: best-firewall-ever + +controllers: + firewall: + enabled: true + + metalAPI: + url: https://metal-api + hmac: some-hmac + type: Metal-View + projectID: abc ``` ### Non-Goals @@ -163,9 +189,8 @@ spec: - path: /etc/firewall-controller/config.yaml content: | --- - main: - additional: - reports: + name: "" + firewall: {} - path: /etc/firewall-controller/seed.yaml secretRef: name: seed-kubeconfig @@ -210,28 +235,29 @@ metadata: stringData: controllerConfig: | --- - main: - config: - url: ${METAL_API_URL} - hmac: ${METAL_API_HMAC} - type: ${METAL_API_HMAC_TYPE} - additional: - - kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/workload.yaml - components: - - kind: ClusterwideNetworkPolicy - namespace: firewall - - kind: Service - - reports: - - kind: kubernetes - config: - kubeconfigPath: /etc/firewall-controller/workload.yaml - components: - - kind: FirewallMonitor - namespace: firewall - name: firewall-monitor-${CLUSTER_NAME} + name: ${CLUSTER_NAME}-firewall + + controllers: + firewall: + metalAPI: + url: ${METAL_API_URL} + hmac: ${METAL_API_HMAC} + type: ${METAL_API_HMAC_TYPE} + projectID: ${METAL_API_PROJECT_ID} + + selfUpdate: + enabled: true + droptailer: + enabled: true + + service: + kubeconfig: /etc/firewall-controller/workload.yaml + cwnp: + kubeconfig: /etc/firewall-controller/workload.yaml + namespace: firewall + monitor: + kubeconfig: /etc/firewall-controller/workload.yaml + namespace: firewall ``` Here the firewall-controller-config will be referenced by the `MetalStackCluster` as a `Secret`. Please note that the `Secret`s in `userdataContents` will not be fetched and will directly be passed to the `FirewallDeployment`. At first the reconciliation of it in the FCM will fail due to the missing Kubeconfig secret. After the `MetalStackCluster` has been marked as ready, CAPI will create this missing secret. Effectively the firewall and initial control plane node should be created at the same time. From 0606b3113ccbd58687dff93507e39128ef19f33c Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Thu, 3 Apr 2025 15:55:13 +0200 Subject: [PATCH 15/18] refactor: use node network instead of firewall entitiy --- .../src/development/proposals/MEP16/README.md | 110 ++++++------------ 1 file changed, 35 insertions(+), 75 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index c0b0c13495..3c5b0846ef 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -1,49 +1,5 @@ # metal-api as an Alternative Configuration Source for the firewall-controller -TODO: -- list of controllers in firewall controllers - - cwnp - - droptailer - - firewall monitor - - service type load balancer - - firewall entity - - firewall resource - - self update -- attach to node network -- more specific instead of generic config - -```yaml -name: best-firewall-ever - -controllers: - firewall: - # one of kubernetes, metalAPI or static - kubernetes: - kubeconfig: /path/to/seed.yaml - metalAPI: - url: https://metal-api - hmac: some-hmac - type: Metal-View - projectID: abc - static: - egress: [] - ingress: [] - - selfUpdate: - enabled: true - droptailer: - enabled: true - - service: - kubeconfig: /path/to/shoot.yaml - cwnp: - kubeconfig: /path/to/shoot.yaml - namespace: firewall - monitor: - kubeconfig: /path/to/shoot.yaml - namespace: firewall -``` - In the current situation, a firewall as provisioned by metal-stack is a fully immutable entity. Any modifications on the firewall like changing the firewall ruleset must be done _somehow_ by the user – the metal-api and hence metal-stack is not aware of its current state. As part of our [integration with the Gardener project](https://docs.metal-stack.io/stable/overview/kubernetes/#Gardener) we offer a solution called the [firewall-controller](https://github.com/metal-stack/firewall-controller), which is part of our [firewall OS images](https://github.com/metal-stack/metal-images/blob/6318a624861b18a559a9d37299bca5f760eef524/firewall/Dockerfile#L57-L58) and addresses shortcomings of the firewall resource's immutability, which would otherwise be completely impractible to work with. The firewall-controller crashes infinitely if it is not properly configured through the userdata when using the firewall image of metal-stack. @@ -63,12 +19,12 @@ With this MEP we want to describe a way to improve this current situation and al ## Proposal The central idea of this proposal is allowing the firewall-controller to use the metal-api as a configuration source. This should serve as an alternative strategy to the currently used FCM `Firewall` resource based approach in the Gardener use-case. -Updates of the firewall rules of the firewall entity should be possible through the metal-api. +Updates of the firewall rules should be possible through the metal-api. The firewall-controller itself should now be able to decide which of the two main strategies should be used for the base configuration: a kubeconfig or the metal-api. This should be possible through a dedicated _firewall-controller-config_. Using this config will now allow operators to fine-tune the data sources for all of its dynamic configuration tasks independently. -For example the data source of the core firewall rules could be set either from the `Firewall` resource located in the Gardener `Seed` or the metal-api firewall entity, while the CWNPs should be fetched and applied from a given kubeconfig (the `Shoot` Kubeconfig in the Gardener case). +For example the data source of the core firewall rules could be set either from the `Firewall` resource located in the Gardener `Seed` or the metal-apiserver node network entity, while the CWNPs should be fetched and applied from a given kubeconfig (the `Shoot` Kubeconfig in the Gardener case). This configuration file is intended to be injected during firewall creation through the userdata along with potential source connection credentials. ```yaml @@ -117,7 +73,6 @@ name: shoot--abc--cluster-firewall-def controllers: firewall: - kubernetes: kubeconfig: /etc/firewall-controller/seed.yaml @@ -160,12 +115,12 @@ controllers: ### Advantages - Offers a native metal-stack solution that improves managing firewalls for users by adding dynamic reconfiguration through the metal-api - - e.g., in the mini-lab, users can now allocate a machine, then an IP address and announce this IP from the machine without having to re-create the firewall but by adding a firewall rule to the firewall entity. + - e.g., in the mini-lab, users can now allocate a machine, then an IP address and announce this IP from the machine without having to re-create the firewall but by adding a firewall rule to the metal-api. - Improve consistency throughout the API (firewall rules would reflect what is persisted in metal-api). - Other providers like Cluster API can leverage this approach, too. - It can contribute to solving the shoot migration issue (in Cluster API case the `clusterctl move` for firewall objects) - For Gardener takes the seed out of the equation (of which the kubeconfig changes during shoot migration) - - However: Things like egress rules, rate limiting, etc. are currently not part of the firewall entity in the metal-api (these would need to be added to the firewall entity as otherwise there is no feature parity) + - However: Things like egress rules, rate limiting, etc. are currently not part of the firewall or network entity in the metal-api. These would need to be added to one of them. ### Caveats @@ -174,13 +129,9 @@ controllers: ### Firewall Controller Manager -As described, the firewall-controller should allow retrieving the main configuration of the firewall from the metal-api (with view HMAC) as an alternative to the kubeconfig based approach that is used in our Gardener setup. - -In this scenario, the firewall-controller-manager is responsible to update the firewall entity in the metal-api whenever the firewall rules change of the firewall resource. - -Of course the metal-api needs to update firewall rules in the machine allocation spec. Additionally ingress and egress rules need to be added to the firewall entity. +Currently the firewall-controller-manager expects the creators of a `FirewallDeployment` to also override all `spec.userdata` used for ignition or none at all. Because of the overhead in terms of implementation in case an adjustment is needed, this feature isn't actually used. -The desired behavior will be configured in the `FirewallDeployment`. Specifically for the Gardener use case, the generation of the kubeconfig for the firewall-controller to be able to access the `Firewall` resource in the seed cluster will be hidden behind the `generateFirewallControllerKubeconfig` flag. +Instead we'd like to propose `spec.userdataContents` which will replace the old `userdata`-string by a typed data structure. The FCM will do the heavy lifting while the `FirewallDeployment` creator decides what should be configured. ```yaml kind: FirewallDeployment @@ -189,8 +140,7 @@ spec: - path: /etc/firewall-controller/config.yaml content: | --- - name: "" - firewall: {} + firewall: {} # ... - path: /etc/firewall-controller/seed.yaml secretRef: name: seed-kubeconfig @@ -200,15 +150,23 @@ spec: name: shoot-kubeconfig ``` +### Gardener Extension Provider Metal Stack + +With this change the GEPM will be responsible to generate the Seed-Kubeconfig and pass the secret reference to the `FirewallDeployment.spec.userdataContents`. + +Additionally the GEPM should also make sure to apply required firewall rules to the node network through the metal-apiserver.. + ### Cluster API Provider Metal Stack + ![architectural overview](firewall-for-capms-overview.svg) -In Cluster API there are essentially two main clusters: the management cluster and the workload cluster. -Typically a local bootstrap cluster is created in kind which acts as the management cluster. It creates the workload cluster. Thereafter the ownership of the workload cluster is typically moved (using `clusterctl move`) to a different cluster which will then become the management cluster. + +In Cluster API there are essentially two main clusters: the management cluster and the workload cluster while the CAPMS takes in the role of the GEPM. +Typically a local bootstrap cluster is created in KinD which acts as the management cluster. It creates the workload cluster. Thereafter the ownership of the workload cluster is typically moved (using `clusterctl move`) to a different cluster which will then become the management cluster. The new management cluster might actually be the workload cluster itself. -In contrast to Gardener, Cluster API aims to be less opinionated. It is common practice to not install any non-required components or CRDs into the workload cluster by default. Therefore we cannot expect custom resources like `ClusterwideNetworkPolicy` or `FirewallMonitor` to be installed in the workload cluster. Therefore it's the responsibility of the operator to tell [cluster-api-provider-metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack) the kubeconfig for the cluster where these CRDs are installed and defined in. +In contrast to Gardener, Cluster API aims to be less opinionated and minimal. It is common practice to not install any non-required components or CRDs into the workload cluster by default. Therefore we cannot expect custom resources like `ClusterwideNetworkPolicy` or `FirewallMonitor` to be installed in the workload cluster but strongly recommend our users to do it. Therefore it's the responsibility of the operator to tell [cluster-api-provider-metal-stack](https://github.com/metal-stack/cluster-api-provider-metal-stack) the kubeconfig for the cluster where these CRDs are installed and defined in. A viable configuration for a `MetalStackCluster` that generates firewall rules based of `Service` type `LoadBalancer` and `ClusterwideNetworkPolicy` and expects them to be deployed in the workload cluster is shown below. The `FirewallMonitor` will be reported into the same cluster. @@ -218,9 +176,6 @@ metadata: name: ${CLUSTER_NAME} spec: firewallTemplate: - # existing fields omitted - staticRuleSet: [] - userdataContents: - path: /etc/firewall-controller/config.yaml secretName: ${CLUSTER_NAME}-firewall-controller-config @@ -278,19 +233,24 @@ In case this control surfaces as a requirement, it would need to be implemented In general this proposal is not thought to be implemented in one batch. Instead an incremental approach is required. -1. Allow Cluster API to use the FCM (provide immutable firewalls that run without firewall-controller). - - Add `spec.staticRuleSet` to Firewall. - - Add `firewall.metal-stack.io/paused` annotation (managed by CAPMS during move, theoretically useful for Gardener shoot migration as well to avoid shallow deletion). - - Reconcile multiple `FirewallDeployment` resources per namespace. - - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance). - - Add `MetalStackCluster.Spec.FirewallTemplate`. - - Make `MetalStackCluster.Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given. -2. Extend firewall-controller with the configuration file (no different data sources than kubernetes) and let the FCM generate this configuration file along with the rest of the userdata. -3. Add metal-api as configuration source. - - Allow updates of firewall rules in the firewall entity. - - For Cluster API: Let the cluster controller generate the userdata including the configuration for this additional data source. -4. Move to more generic interface for the FCM (remove Gardener coupling) +1. Enhance firewall-controller + - reduce coupling between controllers + - introduce controller config + - abstract module to write into distinct nftable rules for every controller + - implement `firewall.static`, but not `firewall.metalAPI` + - GEPM should set `FirewallDeployment.spec.userdataContents` +2. Allow Cluster API to use the FCM with static ruleset + - Add `firewall.metal-stack.io/paused` annotation (managed by CAPMS during move, theoretically useful for Gardener shoot migration as well to avoid shallow deletion). + - Reconcile multiple `FirewallDeployment` resources per namespace. + - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance). + - Add `MetalStackCluster.Spec.FirewallTemplate`. + - Make `MetalStackCluster.Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given. +3. Add `firewall.metalAPI` as configuration option. + - Allow updates of firewall rules in the metal-apiserver. + - Depends on [MEP-4](../MEP4/README.md) metal-apiserver progress +4. Migrate the GEPM to use `firewall.metalAPI` ## Alternatives Considered - instead of the generic data sources, for each mechanism a kubeconfig could be provided. Though this is not a scalable nor flexible approach. In the end the same internal data structure is still needed. +- store network rules in firewall within the metal-apiserver From fb35e6fa55b86b7ffe9217f6914907af4d49788f Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Thu, 10 Apr 2025 12:58:32 +0200 Subject: [PATCH 16/18] docs: update graphic --- .../development/proposals/MEP16/firewall-for-capms-overview.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg index 88b6019567..cedbebf159 100644 --- a/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg +++ b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg @@ -1,4 +1,4 @@ -
handles traffic
Firewall
Firewall Controller
node-exporter
nftables-exporter
droptailer-server
Workload Cluster
droptailer-client
Configures
Bootstrap or Management Cluster
reconcile
configures
reconcile
Cluster API Provider metal-stack
Metal Stack Cluster CRD
Firewall Deployment CRD
Firewall CRD
Firewall Set CRD
rec
reconcile
reconcile
Firewall Controller Manager
Metal Stack Machine CRD
manages
Operator
Kubeconfig FirewallMonitor
FirewallMonitor CRD
main metal-api
Firewall entity
kubeconfig CWNP
Clusterwide Network Policy CRD
base config
controllerConfig
user-defined
network rules
reports firewall
state
collects dropped packages
controllerConfig
controllerConfig
 \ No newline at end of file +
handles traffic
Firewall
Firewall Controller
node-exporter
nftables-exporter
droptailer-client
Workload Cluster
droptailer
Configures
Bootstrap or Management Cluster
reconcile
configures
reconcile
Cluster API Provider metal-stack
Metal Stack Cluster CRD
Firewall Deployment CRD
Firewall CRD
Firewall Set CRD
rec
reconcile
reconcile
Firewall Controller Manager
Metal Stack Machine CRD
manages
Welkin Admin
Kubeconfig FirewallMonitor
FirewallMonitor CRD
main metal-api
Firewall entity
kubeconfig CWNP
Clusterwide Network Policy CRD
base config
controllerConfig
user-defined
network rules
reports firewall
state
send firewall log lines
controllerConfig
controllerConfig
 \ No newline at end of file From a1b04298191c3c3def242b491127ced35cbd3250 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Fri, 11 Apr 2025 09:53:29 +0200 Subject: [PATCH 17/18] refactor: use new sources and reference format --- .../src/development/proposals/MEP16/README.md | 153 ++++++++++++------ 1 file changed, 105 insertions(+), 48 deletions(-) diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 3c5b0846ef..57d0e2eb28 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -31,23 +31,33 @@ This configuration file is intended to be injected during firewall creation thro # the name of the firewall, defaulted to the hostname name: best-firewall-ever +sources: + # exactly one of kubernetes, metal or static + seed: + kubeconfig: /path/to/seed.yaml # current gardener behavior + namespace: shoot--proj--name + shoot: + kubeconfig: /path/to/shoot.yaml # current gardener behavior + namespace: firewall + metal: + url: https://metal-api + hmac: some-hmac + type: Metal-View + projectID: abc + static: + # static should mirror all information provided by the metal or seed/shoot sources + firewall: # optional + controllerURL: https://... + cwnp: + egress: [] + ingress: [] + # all sub-controllers running on the firewall # each can be configured independently controllers: # this is the base controller firewall: - - # one of kubernetes, metalAPI or static - kubernetes: - kubeconfig: /path/to/seed.yaml # current gardener behavior - metalAPI: - url: https://metal-api - hmac: some-hmac - type: Metal-View - projectID: abc - static: - egress: [] - ingress: [] + source: seed # or: metal, static # these are optional: when not provided, they are disabled selfUpdate: @@ -57,38 +67,44 @@ controllers: # these are optional: when not provided, they are disabled service: - kubeconfig: /path/to/shoot.yaml + source: shoot # or: metal, static cwnp: - kubeconfig: /path/to/shoot.yaml - namespace: firewall + source: shoot # or: metal, static monitor: - kubeconfig: /path/to/shoot.yaml - namespace: firewall + source: shoot # currently only shoot is supported ``` +The existing behavior of the firewall-controller writing into `/etc/nftables/firewall-controller.v4` is not changed. The different controller configuration sources are internally treated in the same way as before. The `static` source can be used to prevent the firewall-controller from crashing and consistently providing a static ruleset. This might be interesting for metal-stack native use cases or environments where the metal-api cannot be accessed. + +There must be one central nftables-rule-file-controller that is notified and triggered by all other controllers that contribute to the nftables configuration. + For example, in order to maintain the existing Gardener integration, the configuration file for the firewall-controller will look like this: ```yaml name: shoot--abc--cluster-firewall-def +sources: + seed: + kubeconfig: /etc/firewall-controller/seed.yaml + namespace: shoot--abc--cluster + shoot: + kubeconfig: /etc/firewall-controller/shoot.yaml + namespace: firewall controllers: firewall: - kubernetes: - kubeconfig: /etc/firewall-controller/seed.yaml - + source: seed + selfUpdate: enabled: true droptailer: enabled: true service: - kubeconfig: /etc/firewall-controller/shoot.yaml + source: shoot cwnp: - kubeconfig: /etc/firewall-controller/shoot.yaml - namespace: firewall + source: shoot monitor: - kubeconfig: /etc/firewall-controller/shoot.yaml - namespace: firewall + source: shoot ``` Plain metal-stack users might use a configuration like this: @@ -96,15 +112,51 @@ Plain metal-stack users might use a configuration like this: ```yaml name: best-firewall-ever +sources: + metal: + url: https://metal-api + hmac: some-hmac + type: Metal-View + projectID: abc + shoot: + kubeconfig: /etc/firewall-controller/shoot.yaml + namespace: firewall + +controllers: + firewall: + source: metal + selfUpdate: + enabled: true + droptailer: + enabled: true + + service: + source: shoot + cwnp: + source: shoot + monitor: + source: shoot +``` + +In highly restricted environments that cannot access metal-api the static source could be used: + +```yaml +name: most-restricted-firewall-ever + +sources: + static: + firewall: + controllerURL: https://... + cwnp: + egress: [] + ingress: [] + controllers: firewall: - enabled: true + source: static - metalAPI: - url: https://metal-api - hmac: some-hmac - type: Metal-View - projectID: abc + cwnp: + source: static ``` ### Non-Goals @@ -140,7 +192,11 @@ spec: - path: /etc/firewall-controller/config.yaml content: | --- - firewall: {} # ... + sources: + static: {} + controllers: + firewall: + source: static - path: /etc/firewall-controller/seed.yaml secretRef: name: seed-kubeconfig @@ -158,10 +214,8 @@ Additionally the GEPM should also make sure to apply required firewall rules to ### Cluster API Provider Metal Stack - ![architectural overview](firewall-for-capms-overview.svg) - In Cluster API there are essentially two main clusters: the management cluster and the workload cluster while the CAPMS takes in the role of the GEPM. Typically a local bootstrap cluster is created in KinD which acts as the management cluster. It creates the workload cluster. Thereafter the ownership of the workload cluster is typically moved (using `clusterctl move`) to a different cluster which will then become the management cluster. The new management cluster might actually be the workload cluster itself. @@ -192,27 +246,30 @@ stringData: --- name: ${CLUSTER_NAME}-firewall + sources: + metal: + url: ${METAL_API_URL} + hmac: ${METAL_API_HMAC} + type: ${METAL_API_HMAC_TYPE} + projectID: ${METAL_API_PROJECT_ID} + shoot: + kubeconfig: /etc/firewall-controller/workload.yaml + namespace: firewall + controllers: firewall: - metalAPI: - url: ${METAL_API_URL} - hmac: ${METAL_API_HMAC} - type: ${METAL_API_HMAC_TYPE} - projectID: ${METAL_API_PROJECT_ID} - + source: metal selfUpdate: enabled: true droptailer: enabled: true service: - kubeconfig: /etc/firewall-controller/workload.yaml + source: shoot cwnp: - kubeconfig: /etc/firewall-controller/workload.yaml - namespace: firewall + source: shoot monitor: - kubeconfig: /etc/firewall-controller/workload.yaml - namespace: firewall + source: shoot ``` Here the firewall-controller-config will be referenced by the `MetalStackCluster` as a `Secret`. Please note that the `Secret`s in `userdataContents` will not be fetched and will directly be passed to the `FirewallDeployment`. At first the reconciliation of it in the FCM will fail due to the missing Kubeconfig secret. After the `MetalStackCluster` has been marked as ready, CAPI will create this missing secret. Effectively the firewall and initial control plane node should be created at the same time. @@ -237,7 +294,7 @@ In general this proposal is not thought to be implemented in one batch. Instead - reduce coupling between controllers - introduce controller config - abstract module to write into distinct nftable rules for every controller - - implement `firewall.static`, but not `firewall.metalAPI` + - implement `sources.static`, but not `sources.metal` - GEPM should set `FirewallDeployment.spec.userdataContents` 2. Allow Cluster API to use the FCM with static ruleset - Add `firewall.metal-stack.io/paused` annotation (managed by CAPMS during move, theoretically useful for Gardener shoot migration as well to avoid shallow deletion). @@ -245,10 +302,10 @@ In general this proposal is not thought to be implemented in one batch. Instead - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance). - Add `MetalStackCluster.Spec.FirewallTemplate`. - Make `MetalStackCluster.Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given. -3. Add `firewall.metalAPI` as configuration option. +3. Add `sources.metal` as configuration option. - Allow updates of firewall rules in the metal-apiserver. - Depends on [MEP-4](../MEP4/README.md) metal-apiserver progress -4. Migrate the GEPM to use `firewall.metalAPI` +4. Migrate the GEPM to use `sources.metal` ## Alternatives Considered From c5f973b068106fed8308c02f0dbd79b50b9413a7 Mon Sep 17 00:00:00 2001 From: Valentin Knabel Date: Thu, 17 Apr 2025 12:11:36 +0200 Subject: [PATCH 18/18] fix: typos and wording --- .../src/development/proposals/MEP16/README.md | 95 +++++++++---------- .../firewall-for-capms-overview.drawio.svg | 4 + .../MEP16/firewall-for-capms-overview.svg | 4 - 3 files changed, 51 insertions(+), 52 deletions(-) create mode 100644 docs/src/development/proposals/MEP16/firewall-for-capms-overview.drawio.svg delete mode 100644 docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg diff --git a/docs/src/development/proposals/MEP16/README.md b/docs/src/development/proposals/MEP16/README.md index 57d0e2eb28..68eceed075 100644 --- a/docs/src/development/proposals/MEP16/README.md +++ b/docs/src/development/proposals/MEP16/README.md @@ -32,7 +32,6 @@ This configuration file is intended to be injected during firewall creation thro name: best-firewall-ever sources: - # exactly one of kubernetes, metal or static seed: kubeconfig: /path/to/seed.yaml # current gardener behavior namespace: shoot--proj--name @@ -118,9 +117,6 @@ sources: hmac: some-hmac type: Metal-View projectID: abc - shoot: - kubeconfig: /etc/firewall-controller/shoot.yaml - namespace: firewall controllers: firewall: @@ -130,12 +126,12 @@ controllers: droptailer: enabled: true - service: - source: shoot cwnp: - source: shoot - monitor: - source: shoot + # firewall rules stored in firewall entity + # potential improvement would be to attach the rules to the node network entity + # be aware that the firewall and private networks are immutable + # eventually we introduce a firewall ruleset entity + source: metal ``` In highly restricted environments that cannot access metal-api the static source could be used: @@ -161,7 +157,7 @@ controllers: ### Non-Goals -- Resolving the missing differentiation between users and administrators by lettings users pass userdata and SSH keys to the firewall creation. +- Resolving the missing differentiation between users and administrators by letting users pass userdata and SSH keys to the firewall creation. - This is even more related to [MEP-4](../MEP4/README.md) than this MEP. ### Advantages @@ -173,6 +169,8 @@ controllers: - It can contribute to solving the shoot migration issue (in Cluster API case the `clusterctl move` for firewall objects) - For Gardener takes the seed out of the equation (of which the kubeconfig changes during shoot migration) - However: Things like egress rules, rate limiting, etc. are currently not part of the firewall or network entity in the metal-api. These would need to be added to one of them. +- Potentially resolve the issue that end-users can manipulate accounting data of the firewall through the `FirewallMonitor` + - for this we would need to be able to report traffic data to metal-api ### Caveats @@ -181,40 +179,40 @@ controllers: ### Firewall Controller Manager -Currently the firewall-controller-manager expects the creators of a `FirewallDeployment` to also override all `spec.userdata` used for ignition or none at all. Because of the overhead in terms of implementation in case an adjustment is needed, this feature isn't actually used. +Currently the firewall-controller-manager expects the creators of a `FirewallDeployment` to use the defaulting webhook that is tailored to the Gardener integration in order to generate `Firewall.spec.userdata` or to override it manually. Currently `Firewall.spec.userdata` will never be set explicitly. -Instead we'd like to propose `spec.userdataContents` which will replace the old `userdata`-string by a typed data structure. The FCM will do the heavy lifting while the `FirewallDeployment` creator decides what should be configured. +Instead we'd like to propose `Firewall.spec.userdataContents` which will replace the old `userdata`-string by a typed data structure. The FCM will do the heavy lifting while the `FirewallDeployment` creator decides what should be configured. ```yaml kind: FirewallDeployment spec: - userdataContents: - - path: /etc/firewall-controller/config.yaml - content: | - --- - sources: - static: {} - controllers: - firewall: - source: static - - path: /etc/firewall-controller/seed.yaml - secretRef: - name: seed-kubeconfig - generateFirewallControllerKubeconfig: true - - path: /etc/firewall-controller/shoot.yaml - secretRef: - name: shoot-kubeconfig + template: + spec: + userdataContents: + - path: /etc/firewall-controller/config.yaml + content: | + --- + sources: + static: {} + controllers: + firewall: + source: static + - path: /etc/firewall-controller/seed.yaml + secretRef: + name: seed-kubeconfig + generateFirewallControllerKubeconfig: true + - path: /etc/firewall-controller/shoot.yaml + secretRef: + name: shoot-kubeconfig ``` ### Gardener Extension Provider Metal Stack -With this change the GEPM will be responsible to generate the Seed-Kubeconfig and pass the secret reference to the `FirewallDeployment.spec.userdataContents`. - -Additionally the GEPM should also make sure to apply required firewall rules to the node network through the metal-apiserver.. +The GEPM should be migrated to the new `Firewall.spec.userdataContents` field. ### Cluster API Provider Metal Stack -![architectural overview](firewall-for-capms-overview.svg) +![architectural overview](firewall-for-capms-overview.drawio.svg) In Cluster API there are essentially two main clusters: the management cluster and the workload cluster while the CAPMS takes in the role of the GEPM. Typically a local bootstrap cluster is created in KinD which acts as the management cluster. It creates the workload cluster. Thereafter the ownership of the workload cluster is typically moved (using `clusterctl move`) to a different cluster which will then become the management cluster. @@ -291,23 +289,24 @@ In case this control surfaces as a requirement, it would need to be implemented In general this proposal is not thought to be implemented in one batch. Instead an incremental approach is required. 1. Enhance firewall-controller - - reduce coupling between controllers - - introduce controller config - - abstract module to write into distinct nftable rules for every controller - - implement `sources.static`, but not `sources.metal` - - GEPM should set `FirewallDeployment.spec.userdataContents` + + - Reduce coupling between controllers + - Introduce controller config + - Abstract module to write into distinct nftable rules for every controller + - Implement `sources.static`, but not `sources.metal` + - GEPM should set `FirewallDeployment.spec.template.spec.userdataContents` + 2. Allow Cluster API to use the FCM with static ruleset - - Add `firewall.metal-stack.io/paused` annotation (managed by CAPMS during move, theoretically useful for Gardener shoot migration as well to avoid shallow deletion). - - Reconcile multiple `FirewallDeployment` resources per namespace. - - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance). - - Add `MetalStackCluster.Spec.FirewallTemplate`. - - Make `MetalStackCluster.Spec.NodeNetworkID` optional if `Spec.FirewallTemplate` given. + + - Add `firewall.metal-stack.io/paused` annotation (managed by CAPMS during `clusterctl move`, theoretically useful for Gardener shoot migration as well to avoid shallow deletion). + - Reconcile multiple `FirewallDeployment` resources across multiple namespaces. For Gardener the old behavior of reconciling only one namespace should persist. + - Allow setting the `firewall.metal-stack.io/no-controller-connection` annotation through the `FirewallDeployment` (either through the template or inheritance). + - Add `MetalStackCluster.spec.firewallTemplate`. + - Make `MetalStackCluster.spec.nodeNetworkID` optional if `spec.firewallTemplate` given. + 3. Add `sources.metal` as configuration option. - - Allow updates of firewall rules in the metal-apiserver. - - Depends on [MEP-4](../MEP4/README.md) metal-apiserver progress -4. Migrate the GEPM to use `sources.metal` -## Alternatives Considered + - Allow updates of firewall rules in the metal-apiserver. + - Depends on [MEP-4](../MEP4/README.md) metal-apiserver progress -- instead of the generic data sources, for each mechanism a kubeconfig could be provided. Though this is not a scalable nor flexible approach. In the end the same internal data structure is still needed. -- store network rules in firewall within the metal-apiserver +4. Potentially migrate the GEPM to use `sources.metal` diff --git a/docs/src/development/proposals/MEP16/firewall-for-capms-overview.drawio.svg b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.drawio.svg new file mode 100644 index 0000000000..faea3e3d2e --- /dev/null +++ b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.drawio.svg @@ -0,0 +1,4 @@ + + + +
handles traffic
Firewall
Firewall Controller
node-exporter
nftables-exporter
droptailer-client
Workload Cluster
droptailer
Configures
Bootstrap or Management Cluster
reconcile
configures
reconcile
Cluster API Provider metal-stack
Metal Stack Cluster CRD
Firewall Deployment CRD
Firewall CRD
Firewall Set CRD
rec
reconcile
reconcile
Firewall Controller Manager
Metal Stack Machine CRD
manages
Admin
Kubeconfig FirewallMonitor
FirewallMonitor CRD
main metal-api
Firewall entity
kubeconfig CWNP
Clusterwide Network Policy CRD
base config
controllerConfig
user-defined
network rules
reports firewall
state
send firewall log lines
controllerConfig
controllerConfig
 \ No newline at end of file diff --git a/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg b/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg deleted file mode 100644 index cedbebf159..0000000000 --- a/docs/src/development/proposals/MEP16/firewall-for-capms-overview.svg +++ /dev/null @@ -1,4 +0,0 @@ - - - -
handles traffic
Firewall
Firewall Controller
node-exporter
nftables-exporter
droptailer-client
Workload Cluster
droptailer
Configures
Bootstrap or Management Cluster
reconcile
configures
reconcile
Cluster API Provider metal-stack
Metal Stack Cluster CRD
Firewall Deployment CRD
Firewall CRD
Firewall Set CRD
rec
reconcile
reconcile
Firewall Controller Manager
Metal Stack Machine CRD
manages
Welkin Admin
Kubeconfig FirewallMonitor
FirewallMonitor CRD
main metal-api
Firewall entity
kubeconfig CWNP
Clusterwide Network Policy CRD
base config
controllerConfig
user-defined
network rules
reports firewall
state
send firewall log lines
controllerConfig
controllerConfig
 \ No newline at end of file