Support DualStack load balancer in GatewayAPI service

[sebhoss]

We have a dual-stack cluster and want to set the ipFamilyPolicy of the generated Service to RequireDualStack. Currently that is not possible and the operator always sets this to SingleStack which allocates only an IPv4 address in our stack. Something like the following would be perfect:

apiVersion: operator.tigera.io/v1
kind: GatewayAPI
metadata:
  name: default
spec:
  gatewayClasses:
    - name: calico
      gatewayService:
        spec:
          ipFamilyPolicy: RequireDualStack

[nelljerram]

@sebhoss Can I just check which Service you mean? The Service for the gateway controller, or the Service that gets provisioned for each Gateway?

This kind of request makes sense to me, but I just want to be sure I understand it in detail.

[nelljerram]

D'oh, I realized after reading #4718 as well that you obviously mean the per-Gateway Service.

[nelljerram]

So the situation is the same as @electricjesus said here. Envoy's API doesn't currently expose a direct way to set ipFamilyPolicy in the Services that it creates - i.e. in https://gateway.envoyproxy.io/latest/api/extension_types/#kubernetesservicespec - so it isn't straightforward for us to do this, unless Envoy is enhanced first.

Except ... if it's possible using the patch field in KubernetesServiceSpec. If you're interested in pursuing this immediately, you could try:

1. Configure a Gateway in Calico as things work today.
2. Record the resulting EnvoyProxy YAML (named after the gateway class, in namespace "tigera-gateway").
3. Create your own EnvoyProxy based on that YAML, but with KubernetesServiceSpec.patch crafted to set ipFamilyPolicy in the way you want.
4. Configure a GatewayClass that uses the envoyProxyRef field to reference your EnvoyProxy.
5. Configure a Gateway with that class, and check the results.
6. Iterate if needed from (3).

If you're able to craft a patch that works, we could then look at adding a field in our GatewayServiceSpec struct and plumbing it through to your patch.

[nelljerram]

@sebhoss I had a similar kind of conversation back in February where a user reported success with a patch like this:

 apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: EnvoyProxy
 metadata:
   name: custom
   namespace: tigera-gateway
 spec:
   provider:
     kubernetes:
       envoyService:
         patch:
           type: JSONMerge
           value:
             spec:
               ports:
               - name: http-80
                 nodePort: 30008
                 port: 80
                 protocol: TCP
                 targetPort: 10080
               - name: https-443
                 nodePort: 30004
                 port: 443
                 protocol: TCP
                 targetPort: 10443

we could then look at adding an ipFamilyPolicy field in our GatewayServiceSpec struct and plumbing it through to your patch

Or alternatively we could add a general patch field in GatewayServiceSpec that we pass through to KubernetesServiceSpec. Then that would cover both this and #4718 I think.

(cc @electricjesus )

[sebhoss]

thanks @nelljerram & @electricjesus! I've now successfully deployed the following:

1. An EnvoyProxy that patches the ipFamilyPolicy and healthCheckNodePort fields:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: calico-custom
  namespace: tigera-gateway
spec:
  logging:
    level:
      default: warn
  provider:
    kubernetes:
      envoyDaemonSet:
        container:
          image: harbor.infra.run/quay/calico/envoy-proxy:v3.31.5
        pod:
          nodeSelector:
            node-role.kubernetes.io/gateway: ""
          tolerations:
            - effect: NoSchedule
              key: node-role.kubernetes.io/gateway
              operator: Exists
      envoyService:
        allocateLoadBalancerNodePorts: false
        annotations:
          purelb.io/allocated-by: PureLB
          purelb.io/service-group: default
        externalTrafficPolicy: Local
        loadBalancerClass: purelb.io/purelb
        type: LoadBalancer
        patch:
          type: JSONMerge
          value:
            spec:
              ipFamilyPolicy: RequireDualStack
              healthCheckNodePort: 31222
    type: Kubernetes

2. A GatewayClass that references the new EnvoyProxy:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: calico-custom
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: calico-custom
    namespace: tigera-gateway

3. A Gateway that references the new GatewayClass:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: default
  namespace: tigera-gateway
spec:
  gatewayClassName: calico-custom
  listeners:
    - name: http
      hostname: dummy # we are using ListenerSets but need to supply at least one listener here - https://github.com/kubernetes-sigs/gateway-api/issues/4425
      protocol: HTTP
      port: 80

The result is that the Service now gets both an IPv4 and IPv6 address from PureLB and its HCNP is set to a known value.

From a user perspective I think it would be great to be able to pass a general patch in GatewayServiceSpec or just pass these fields directly. The patch approach looks a little more complex for users but it's more powerful and allows greater flexibility, however it kinds classes with how the service is currently being configured since some fields are directly exposed in GatewayServiceSpec? I'll take whatever you guys want to use in the end!