fix(cilium): add dynamic api server endpoint configuration

[r3m8]

What type of PR is this?
/kind bug

What this PR does / why we need it:

This pull request automatically configures Cilium's API server endpoint for high availability (HA) clusters. It patches the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables in both cilium-operator Deployment and cilium DaemonSet to match the cluster's load balancer configuration.

In HA clusters with multiple control planes, Cilium pods would hardcode the first control plane IP, causing complete network failure when that node shuts down. Even when users configured cilium_config_extra_vars with the correct endpoint, the environment variables injected at pod creation time were not updated.

After Cilium installation/upgrade, the playbook now automatically:

1. Detects the appropriate API server endpoint based on cluster configuration
2. Patches both cilium-operator and cilium agent pods with correct environment variables
3. Waits for rollout to complete, ensuring cluster stability

Configuration priority:

1. Explicit override: User-defined cilium_config_extra_vars.k8sServiceHost (highest priority)
2. External load balancer: loadbalancer_apiserver if defined
3. Local load balancer: 127.0.0.1 (default for HA clusters with nginx-proxy)

This makes HA Cilium clusters work correctly out-of-the-box without manual intervention.

Which issue(s) this PR fixes:
Fixes #12623

Special notes for your reviewer:

Playbook log on Gist

Does this PR introduce a user-facing change?:

fix: automatically configure Cilium API server endpoint for HA clusters

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: r3m8 / name: r3m8 (eb8728a)

[k8s-ci-robot]

Welcome @r3m8!

It looks like this is your first PR to kubernetes-sigs/kubespray 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kubespray has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: r3m8
Once this PR has been reviewed and has the lgtm label, please assign mzaian for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @r3m8. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tico88612]

Can you carefully review the existing implementation and integrate it with values.yaml? I don't think we need to maintain another task.

[r3m8]

I will take the time to check this implementation.

[r3m8]

I think I've found the root cause of the issue. By default, it looks like the cilium k8sServiceHost parameter (auto in values.yaml) directly uses the value written in the cluster-info ConfigMap, which itself inherits from the kube_apiserver_endpoint variable.

However, the Cilium ds/rs pull the value from first_kube_control_plane_address:

kube_apiserver_endpoint: |-
  {% if loadbalancer_apiserver is defined -%}
      https://{{ apiserver_loadbalancer_domain_name }}:{{ loadbalancer_apiserver.port | default(kube_apiserver_port) }}
  {%- elif ('kube_control_plane' not in group_names) and loadbalancer_apiserver_localhost -%}
      https://localhost:{{ loadbalancer_apiserver_port | default(kube_apiserver_port) }}
  {%- elif 'kube_control_plane' in group_names -%}
  https://{{ kube_apiserver_bind_address | regex_replace('::', '127.0.0.1') | ansible.utils.ipwrap }}:{{ kube_apiserver_port }}
  {%- else -%}
      https://{{ first_kube_control_plane_address | ansible.utils.ipwrap }}:{{ kube_apiserver_port }}
  {%- endif %}

cm cluster-info:

kubectl get cm cluster-info -n kube-public -o yaml
apiVersion: v1
data:
  jws-kubeconfig-...
  kubeconfig: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: ...
        server: [https://10.230.9.11:6443](https://10.230.9.11:6443)
      name: ""

rs cilium-operator:

    Environment:
      K8S_NODE_NAME:             (v1:spec.nodeName)
      CILIUM_K8S_NAMESPACE:     kube-system (v1:metadata.namespace)
      CILIUM_DEBUG:             <set to the key 'debug' of config map 'cilium-config'>  Optional: true
      KUBERNETES_SERVICE_HOST:  10.230.9.11
      KUBERNETES_SERVICE_PORT:  6443

ds cilium-envoy:

    Environment:
      K8S_NODE_NAME:             (v1:spec.nodeName)
      CILIUM_K8S_NAMESPACE:      (v1:metadata.namespace)
      KUBERNETES_SERVICE_HOST:  10.230.9.11
      KUBERNETES_SERVICE_PORT:  6443

ds cilium:

    Environment:
      K8S_NODE_NAME:             (v1:spec.nodeName)
      CILIUM_K8S_NAMESPACE:      (v1:metadata.namespace)
      KUBERNETES_SERVICE_HOST:  10.230.9.11
      KUBERNETES_SERVICE_PORT:  6443

I think forcing the definition to 127.0.0.1 is the best solution given the current setup on both workers and control planes.

Priority order for KUBERNETES_SERVICE_HOST:

1. cilium_config_extra_vars['k8s-service-host']
2. cilium_config_extra_vars['k8sServiceHost']
3. apiserver_loadbalancer_domain_name (if loadbalancer_apiserver is defined)
4. loadbalancer_apiserver.address (if loadbalancer_apiserver is defined)
5. "127.0.0.1" (if loadbalancer_apiserver_localhost == true)
6. first_kube_control_plane_address (fallback)

And for KUBERNETES_SERVICE_PORT:

1. cilium_config_extra_vars['k8s-service-port']
2. cilium_config_extra_vars['k8sServicePort']
3. loadbalancer_apiserver.port (if loadbalancer_apiserver is defined)
4. loadbalancer_apiserver_port (if loadbalancer_apiserver_localhost == true)
5. kube_apiserver_port (fallback)

[r3m8]

YAML's >- folding operator and Jinja2's whitespace control conflict with each other in values.yaml.j2, causing invalid YAML output. For better readability and maintainability, wouldn't it make more sense to pre-calculate these values in a separate task like this?

I tested both implementations, and they work.

- name: Cilium | Determine API server endpoint for HA
  set_fact:
    cilium_k8s_service_host: >-
      {%- if cilium_config_extra_vars is defined and 'k8s-service-host' in cilium_config_extra_vars -%}
        {{ cilium_config_extra_vars['k8s-service-host'] }}
      {%- elif loadbalancer_apiserver is defined and apiserver_loadbalancer_domain_name is defined -%}
        {{ apiserver_loadbalancer_domain_name }}
      {%- elif loadbalancer_apiserver is defined and loadbalancer_apiserver.address is defined -%}
        {{ loadbalancer_apiserver.address }}
      {%- elif loadbalancer_apiserver_localhost is defined and loadbalancer_apiserver_localhost == true -%}
        127.0.0.1
      {%- else -%}
        {{ first_kube_control_plane_address }}
      {%- endif -%}

    cilium_k8s_service_port: >-
      {%- if cilium_config_extra_vars is defined and 'k8s-service-port' in cilium_config_extra_vars -%}
        {{ cilium_config_extra_vars['k8s-service-port'] }}
      {%- elif loadbalancer_apiserver is defined and loadbalancer_apiserver.port is defined -%}
        {{ loadbalancer_apiserver.port }}
      {%- elif loadbalancer_apiserver_localhost is defined and loadbalancer_apiserver_localhost == true and loadbalancer_apiserver_port is defined -%}
        {{ loadbalancer_apiserver_port }}
      {%- else -%}
        {{ kube_apiserver_port }}
      {%- endif -%}
  when: kube_network_plugin == "cilium"
  tags:
    - cilium
    - network

k8sServiceHost: "{{ cilium_k8s_service_host | default('127.0.0.1') }}"
k8sServicePort: "{{ cilium_k8s_service_port | default('6443') }}"

[VannTen]

wouldn't it make more sense to pre-calculate these values in a separate task like this?

It's probably better to put it in the role vars/ folders instead. (set_fact is rarely justified, even if we have pervasive usage in Kubespray, most of it should go away)

(Not expressing an opinion on the rest, haven't had the time to review more thoroughly)

[r3m8]

It's probably better to put it in the role vars/ folders instead.

Done. I've successfully rebased on upstream.

Pre-commit tests are passing, I've squashed into 1 commit and properly renamed the PR.

$ kubectl logs -n kube-system cilium-operator-5b9fc5b548
time=2025-12-02T17:27:45.359620009Z level=info msg="Cilium Operator" subsys=cilium-operator-generic version="1.18.4 afda2aa9 2025-11-12T10:14:04+00:00 go version go1.24.10 linux/amd64"
time=2025-12-02T17:27:45.397054203Z level=info source=/go/src/github.com/cilium/cilium/pkg/gops/cell.go:48 msg="Started gops server" module=operator.operator-infra.gops address=127.0.0.1:9891
time=2025-12-02T17:27:45.418093392Z level=info source=/go/src/github.com/cilium/cilium/pkg/k8s/client/cell.go:317 msg="Establishing connection to apiserver" module=operator.operator-infra.operator-k8s-client-builder.k8s-client-builder ipAddr=https://127.0.0.1:6443

$ kubectl get rs -n kube-system cilium-operator-5b9fc5b548-ldtmk
[...]
        - name: KUBERNETES_SERVICE_HOST
          value: 127.0.0.1
        - name: KUBERNETES_SERVICE_PORT
          value: "6443"

$ kubectl get ds -n kube-system cilium-envoy -o yaml
[...]
        - name: KUBERNETES_SERVICE_HOST
          value: 127.0.0.1
        - name: KUBERNETES_SERVICE_PORT
          value: "6443"

$ kubectl get ds -n kube-system cilium-envoy -o yaml
[...]
        - name: KUBERNETES_SERVICE_HOST
          value: 127.0.0.1
        - name: KUBERNETES_SERVICE_PORT
          value: "6443"

[VannTen]

The config extra vars stuff is passed as a second values files to ciliumcli right ? In that case you don't need to check it in the default value, since it will be override at helm level and the value rendered in the first value file will be irrelevant. Also, I think you could use `kube_apiserver_global_endpoint` or whatever the name is, check a similar thing in #12598

[r3m8]

The config extra vars stuff is passed as a second values files to ciliumcli right ?

Yes, Helm will override via the second values file anyway.

Also, I think you could use kube_apiserver_global_endpoint

I agree. I replicated Calico's current configuration. I was able to launch the deployment, and it works.

[r3m8]

In this case, do you still prefer to use role vars/folders, or can we place this directly in templates/values.yaml.j2 like this ? Calico already use this pattern.

k8sServiceHost: "{{ kube_apiserver_global_endpoint | urlsplit('hostname') }}"

k8sServicePort: "{{ kube_apiserver_global_endpoint | urlsplit('port') }}"

[VannTen]

Yeah, eschew the variables, the indirection only adds complexity for a simple expression like this.

[r3m8]

It's done, we're good.

[tico88612]

/ok-to-test

[VannTen]

This seems sane to me, but I'm not deep in cilium.
@tico88612 changes makes sense to you ?

[tico88612]

/cc

I'll review later.