Hcp tentant example #147
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| global | ||
| log 127.0.0.1 local2 | ||
| pidfile /var/run/haproxy.pid | ||
| maxconn 4000 | ||
| daemon | ||
| defaults | ||
| mode http | ||
| log global | ||
| option dontlognull | ||
| option http-server-close | ||
| option redispatch | ||
| retries 3 | ||
| timeout http-request 10s | ||
| timeout queue 1m | ||
| timeout connect 10s | ||
| timeout client 1m | ||
| timeout server 1m | ||
| timeout http-keep-alive 10s | ||
| timeout check 10s | ||
| maxconn 3000 | ||
|
|
||
| listen api | ||
| bind *:6443 | ||
| mode tcp | ||
| balance source | ||
| server ucs-blade-server-5 10.32.96.105:30918 check inter 1s | ||
| server ucs-blade-server-6 10.32.96.106:30918 check inter 1s | ||
| server ucs-blade-server-7 10.32.96.107:30918 check inter 1s | ||
| server ucs-blade-server-8 10.32.96.108:30918 check inter 1s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,277 @@ | ||
| --- | ||
| title: Hosted Control Plane and tenant networking | ||
| linktitle: Hosted Control Plane and tenant networking | ||
| description: Hosted Control Plane and tenant networking | ||
| tags: ['hcp','v4.21'] | ||
| --- | ||
| # Hosted Control Plane and tenant networking | ||
|
|
||
| Official documentation: Not yet available | ||
|
|
||
| Tested with: | ||
|
|
||
| |Component|Version| | ||
| |---|---| | ||
| |OpenShift|v4.21.9| | ||
| |OpenShift Virt|v4.21.0| | ||
|
|
||
| ## Overview | ||
|
|
||
| Challenge: running a hosted cluster in a different tenant network segment or VLAN without wide-open access from the tenant segment to the management segment. | ||
|
|
||
| Additional requirement: the hub cluster must not have arbitrary addressing or routing into the tenant network segment. The hub may only attach hosted-cluster workloads (for example, KubeVirt VMs) to that segment. | ||
|
|
||
| { page="Page-1" } | ||
|
|
||
| Worker nodes are straightforward: attach them to the tenant network segment (DHCP or equivalent addressing is required). | ||
|
|
||
| Exposing hosted control plane endpoints into the tenant segment is harder. The following components must be reachable from workers and clients in that segment: | ||
|
|
||
| * API Server | ||
| * OAuth | ||
| * Konnectivity | ||
| * Ignition | ||
|
|
||
| Here is a summary of common publishing options for these components: | ||
|
|
||
| |Component/Service|Exposing strategy (`servicePublishingStrategy`)|Kubernetes Service type `LoadBalancer`|Route (OpenShift router)| | ||
| |---|---|---|---| | ||
| |API Server|<li>LoadBalancer (recommended; Kubernetes `LoadBalancer` service)</li><li>NodePort* (not for production)</li>|✅|❌| | ||
| |OAuth|<li>Route (default)</li><li>NodePort* (not for production)</li>|❌|✅| | ||
| |Konnectivity|<li>Route (default)</li><li>LoadBalancer (Kubernetes `LoadBalancer` service)</li><li>NodePort* (not for production)</li>|✅|✅| | ||
| |Ignition|<li>Route (default)</li><li>NodePort* (not for production)</li>|✅|❌| | ||
|
|
||
| For this proof of concept, endpoints are exposed as follows: | ||
|
|
||
| * API Server: `LoadBalancer` (fronted by external `api-lb` in the tenant segment; see below) | ||
| * OAuth, Konnectivity, Ignition: `Route` via a **dedicated ingress controller shard** on the hub, fronted by external `ingress-shared-lb` with VIPs/DNS in the tenant segment | ||
|
|
||
| ## Exposing components via a dedicated router shard | ||
|
|
||
| Use a dedicated OpenShift Ingress Controller shard on the **hub** so only the hosted-cluster control-plane Routes are served by that shard. Tenant clients resolve OAuth, Konnectivity, and Ignition hostnames to `ingress-shared-lb`, which forwards to the shard’s NodePorts on the management network. | ||
|
Collaborator
There was a problem hiding this comment. I wonder why |
||
|
|
||
| Place an external load balancer in front of that shard (for example F5 BIG-IP or NetScaler) that can reach the hub’s management network and present stable tenant-facing VIPs or addresses. | ||
|
Collaborator
There was a problem hiding this comment. NIT: (for example F5, BIG-IP or NetScaler) |
||
|
|
||
| ## Proof of concept environment overview | ||
|
|
||
| { page="Page-2" } | ||
|
|
||
| ### Router between Mgmt and Tenant-A | ||
|
|
||
| [VyOS](https://vyos.io/) acts as router and firewall between management and Tenant-A. Restrict **lateral** traffic between the two segments (no full mesh); allow only what you need (for example DNS to resolvers, default route or NAT for internet egress). Hosted-cluster control-plane traffic from tenant nodes should flow to the **external load balancer VIPs** in the tenant segment (not directly into arbitrary management subnets). | ||
|
Collaborator
There was a problem hiding this comment. NIT: VyOS acts as router and firewall between the management and the Tenant-A network. |
||
|
|
||
| ??? example "VyOS config commands" | ||
|
|
||
| ```shell | ||
| --8<-- "content/cluster-installation/hosted-control-plane/tenant-network/vyos-router-2003.txt" | ||
| ``` | ||
|
|
||
| ### Ingress Sharding at Hub Cluster | ||
|
|
||
| * [2.3.4. Ingress sharding in OpenShift Container Platform](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/ingress_and_load_balancing/configuring-ingress-cluster-traffic#nw-ingress-sharding-concept_configuring-ingress-cluster-traffic-ingress-controller) | ||
| * [3.1.3.8.1. Example load balancer configuration for user-provisioned clusters](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/installing_on_vmware_vsphere/user-provisioned-infrastructure) | ||
|
|
||
| ???+ example "Ingress Controller" | ||
|
Collaborator
There was a problem hiding this comment. As already commented above. Why oc create -f - <<EOF
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: sharded-hcp-2003-57
namespace: openshift-ingress-operator
spec:
routeAdmission:
wildcardPolicy: WildcardsAllowed
domain: hosted.hcp-2003-57.hcp-cps.coe.muc.redhat.com
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: clusters-hcp-2003-57
endpointPublishingStrategy:
type: LoadBalancerService
EOF |
||
|
|
||
| ```yaml | ||
| --8<-- "content/cluster-installation/hosted-control-plane/tenant-network/ingress-controller-shard.yaml" | ||
| ``` | ||
|
|
||
| ```shell | ||
| % oc get svc -n openshift-ingress router-nodeport-tenant-a | ||
| NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE | ||
| router-nodeport-tenant-a NodePort 172.30.141.209 <none> 80:32460/TCP,443:32488/TCP,1936:32095/TCP 106s | ||
| ``` | ||
|
|
||
| The ingress shard load balancer is an RHEL 9 host running HAProxy (external load balancer `ingress-shared-lb`). | ||
|
Collaborator
There was a problem hiding this comment. I think this will be the answer to my |
||
|
|
||
| * Install HAProxy: `dnf install haproxy` | ||
| * Configure SELinux: `setsebool -P haproxy_connect_any 1` | ||
| * Apply the example `haproxy` configuration (update ports to match your NodePort service) | ||
| * Enable and start HAProxy: `systemctl enable --now haproxy` | ||
|
|
||
| ??? example "HAProxy config" | ||
|
|
||
| ```shell | ||
| --8<-- "content/cluster-installation/hosted-control-plane/tenant-network/ingress-shared-haproxy.conf" | ||
| ``` | ||
|
|
||
| Add DNS records | ||
|
|
||
| ```bind | ||
| konnectivity.tenant-a.coe.muc.redhat.com. IN A 192.168.203.111 | ||
| oauth.tenant-a.coe.muc.redhat.com. IN A 192.168.203.111 | ||
| ignition.tenant-a.coe.muc.redhat.com. IN A 192.168.203.111 | ||
| ``` | ||
|
|
||
| ### Deployment sequence (reference) | ||
|
|
||
| Three external load balancers appear in this write-up; keep their roles distinct: | ||
|
|
||
| | Name | Role | | ||
| |------|------| | ||
| | `ingress-shared-lb` | Tenant-facing VIPs for OAuth, Konnectivity, Ignition Routes on the **hub** ingress shard | | ||
| | `api-lb` | Tenant-facing VIP for the hosted cluster **API** (`APIServer` publishing) | | ||
| | `ingress-lb` | Tenant-facing VIP for **hosted cluster** application Routes (`*.apps…`) | | ||
|
|
||
| Suggested order: (1) hub ingress shard + `ingress-shared-lb` + DNS for the three control-plane hostnames, (2) `api-lb` + API DNS, (3) `ingress-lb` + wildcard apps DNS, then (4) apply `HostedCluster` and `NodePool`. Adjust if your automation creates services first and you backfill DNS once NodePorts or service endpoints are known. | ||
|
Collaborator
There was a problem hiding this comment. Make it a list for better readability. Furthermore, initially I found it hard to understand what is meant by "Suggested order". You mean the configuration/deployment order? |
||
|
|
||
| The following two subsections describe (2) and (3); the hub shard and DNS for OAuth, Konnectivity, and Ignition are covered above. | ||
|
|
||
| ### Deploy External Load Balancer for API (`api-lb`) | ||
|
Collaborator
There was a problem hiding this comment. Maybe rephrase to "Deploy External Load Balancer for the Hosted-Cluster API ( |
||
|
|
||
| Use an RHEL 9 virtual machine with HAProxy. | ||
|
|
||
| * Install HAProxy: `dnf install haproxy` | ||
| * Configure SELinux: `setsebool -P haproxy_connect_any 1` | ||
| * Apply the example `haproxy` configuration (update ports to match your environment) | ||
| * Enable and start HAProxy: `systemctl enable --now haproxy` | ||
|
|
||
| ??? example "HAProxy config" | ||
|
|
||
| ```shell | ||
| --8<-- "content/cluster-installation/hosted-control-plane/tenant-network/api-lb.conf" | ||
| ``` | ||
|
|
||
| Add DNS record: | ||
|
|
||
| ```bind | ||
| api.tenant-a.coe.muc.redhat.com. IN A 192.168.203.<IP of VM> | ||
| ``` | ||
|
|
||
| ### Deploy External Load Balancer for Ingress (`ingress-lb`) of hosted cluster | ||
|
Collaborator
There was a problem hiding this comment. Maybe rephrase to "Deploy External Load Balancer for the Hosted-Cluster Ingress ( |
||
|
|
||
| Use an RHEL 9 virtual machine with HAProxy. | ||
|
|
||
| * Install HAProxy: `dnf install haproxy` | ||
| * Configure SELinux: `setsebool -P haproxy_connect_any 1` | ||
| * Apply the example `haproxy` configuration (update ports to match your environment) | ||
| * Enable and start HAProxy: `systemctl enable --now haproxy` | ||
|
|
||
| ??? example "HAProxy config" | ||
|
|
||
| ```shell | ||
| --8<-- "content/cluster-installation/hosted-control-plane/tenant-network/ingress-lb.conf" | ||
| ``` | ||
|
|
||
| Add DNS record: | ||
|
|
||
| ```bind | ||
| *.apps.tenant-a.coe.muc.redhat.com. IN A 192.168.203.<IP of VM> | ||
| ``` | ||
|
|
||
| ### Start hosted control plane and nodepool | ||
|
|
||
| ```yaml hl_lines="11 43-66" title="HostedCluster" | ||
| apiVersion: hypershift.openshift.io/v1beta1 | ||
| kind: HostedCluster | ||
| metadata: | ||
| name: 'tenant-a' | ||
| namespace: 'clusters' | ||
| labels: | ||
| "cluster.open-cluster-management.io/clusterset": 'default' | ||
| spec: | ||
| configuration: | ||
| ingress: | ||
| appsDomain: apps.tenant-a.coe.muc.redhat.com # (1) | ||
| domain: '' | ||
| loadBalancer: | ||
| platform: | ||
| type: '' | ||
| channel: fast-4.21 | ||
| etcd: | ||
| managed: | ||
| storage: | ||
| persistentVolume: | ||
| size: 8Gi | ||
| type: PersistentVolume | ||
| managementType: Managed | ||
| release: | ||
| image: quay.io/openshift-release-dev/ocp-release:4.21.11-multi | ||
| pullSecret: | ||
| name: pullsecret-cluster-tenant-a | ||
| sshKey: | ||
| name: sshkey-cluster-tenant-a | ||
| networking: | ||
| clusterNetwork: | ||
| - cidr: 10.132.0.0/14 | ||
| serviceNetwork: | ||
| - cidr: 172.31.0.0/16 | ||
| networkType: OVNKubernetes | ||
| controllerAvailabilityPolicy: SingleReplica | ||
| infrastructureAvailabilityPolicy: SingleReplica | ||
| platform: | ||
| type: KubeVirt | ||
| kubevirt: | ||
| baseDomainPassthrough: false | ||
| infraID: 'tenant-a' | ||
| services: | ||
| - service: APIServer | ||
| servicePublishingStrategy: | ||
| type: LoadBalancer | ||
| loadBalancer: | ||
| hostname: api.tenant-a.coe.muc.redhat.com # (2) | ||
| - service: OAuthServer | ||
| servicePublishingStrategy: | ||
| type: Route | ||
| route: | ||
| hostname: oauth.tenant-a.coe.muc.redhat.com # (3) | ||
| - service: OIDC | ||
| servicePublishingStrategy: | ||
| type: Route | ||
| - service: Konnectivity | ||
| servicePublishingStrategy: | ||
| type: Route | ||
| route: | ||
| hostname: konnectivity.tenant-a.coe.muc.redhat.com # (4) | ||
| - service: Ignition | ||
| servicePublishingStrategy: | ||
| type: Route | ||
| route: | ||
| hostname: ignition.tenant-a.coe.muc.redhat.com # (5) | ||
| ``` | ||
|
|
||
| 1. `appsDomain`: resolve names under `apps.tenant-a.coe.muc.redhat.com` to **`ingress-lb`** (hosted cluster ingress), not the hub shard. | ||
| 2. API server `loadBalancer.hostname`: resolve to **`api-lb`**, which forwards to the `APIServer` publishing target on the hub. | ||
| 3. OAuth `route.hostname`: resolve to **`ingress-shared-lb`** (hub dedicated shard). | ||
| 4. Konnectivity `route.hostname`: resolve to **`ingress-shared-lb`**. | ||
| 5. Ignition `route.hostname`: resolve to **`ingress-shared-lb`**. | ||
|
|
||
| ```yaml hl_lines="24-26" title="NodePool" | ||
| apiVersion: hypershift.openshift.io/v1beta1 | ||
| kind: NodePool | ||
| metadata: | ||
| name: 'tenant-a' | ||
| namespace: 'clusters' | ||
| spec: | ||
| arch: amd64 | ||
| clusterName: 'tenant-a' | ||
| replicas: 2 | ||
| management: | ||
| autoRepair: false | ||
| upgradeType: Replace | ||
| platform: | ||
| type: KubeVirt | ||
| kubevirt: | ||
| compute: | ||
| cores: 2 | ||
| memory: 8Gi | ||
| rootVolume: | ||
| type: Persistent | ||
| persistent: | ||
| size: 32Gi | ||
| additionalNetworks: | ||
| - name: default/cudn-localnet1-2003 # (1) | ||
|
Collaborator
There was a problem hiding this comment. Maybe including the creation of the CUDN before this manifest as well`? oc apply -f - <<EOF
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
name: cudn-localnet1-2003
spec:
namespaceSelector:
matchExpressions:
- key: hypershift.openshift.io/hosted-control-plane
operator: Exists
network:
topology: Localnet
localnet:
role: Secondary
physicalNetworkName: localnet1
ipam:
mode: Disabled
vlan:
mode: Access
access:
id: 2003
EOF |
||
| attachDefaultNetwork: false | ||
| release: | ||
| image: quay.io/openshift-release-dev/ocp-release:4.21.11-multi | ||
| ``` | ||
|
|
||
| 1. Attach NodePool VMs to the tenant segment using a user-defined network (UDN) `localnet` attachment (`default/cudn-localnet1-2003` in this lab). | ||
|
|
||
| ## Open topics | ||
|
|
||
| * Disable or constrain cloud provider integration so that Kubernetes `LoadBalancer` Service requests for the hosted cluster are not satisfied by the hub cluster cloud integration unless that is intentional. | ||
| * WebUI bug: ACM shows `https://console-openshift-console.apps.tenant-a.apps.ocp5.stormshift.coe.muc.redhat.com/` for the console, but the URL should be `https://console-openshift-console.apps.tenant-a.coe.muc.redhat.com/`. | ||
| * Add custom endpoint publishing strategy | ||
| * Find a solution for the NodePort chicken-and-egg problem of the external API load balancer | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| apiVersion: operator.openshift.io/v1 | ||
| kind: IngressController | ||
| metadata: | ||
| name: tenant-a | ||
| namespace: openshift-ingress-operator | ||
| spec: | ||
| domain: tenant-a.coe.muc.redhat.com | ||
|
|
||
| endpointPublishingStrategy: | ||
| type: NodePortService | ||
| namespaceSelector: | ||
| matchExpressions: | ||
| - key: kubernetes.io/metadata.name | ||
| operator: In | ||
| values: | ||
| - ingress-test | ||
| - clusters-tenant-a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| global | ||
| log 127.0.0.1 local2 | ||
| pidfile /var/run/haproxy.pid | ||
| maxconn 4000 | ||
| daemon | ||
| defaults | ||
| mode http | ||
| log global | ||
| option dontlognull | ||
| option http-server-close | ||
| option redispatch | ||
| retries 3 | ||
| timeout http-request 10s | ||
| timeout queue 1m | ||
| timeout connect 10s | ||
| timeout client 1m | ||
| timeout server 1m | ||
| timeout http-keep-alive 10s | ||
| timeout check 10s | ||
| maxconn 3000 | ||
|
|
||
| listen ingress-router-443 | ||
| bind *:443 | ||
| mode tcp | ||
| balance source | ||
| server tenant-a-gngj5-mfwp6 192.168.203.101:30190 check inter 1s | ||
| server tenant-a-gngj5-rrbmv 192.168.203.102:30190 check inter 1s | ||
|
|
||
| listen ingress-router-80 | ||
| bind *:80 | ||
| mode tcp | ||
| balance source | ||
| server tenant-a-gngj5-mfwp6 192.168.203.101:30282 check inter 1s | ||
| server tenant-a-gngj5-rrbmv 192.168.203.102:30282 check inter 1s |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ✅ is set for LoadBalancer but must be
Route.