CKS: Fix issue with scaling down CKS Nodes when deployed in HA mode

[Pearl1594]

Description

This PR fixes: #12229
Fixes issue when scaling down k8s clusters deployed in HA mode - indexing issue encountered.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

❌ Patch coverage is 62.50000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.59%. Comparing base (8936e4c) to head (dd07d7d).
⚠️ Report is 12 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...er/actionworkers/KubernetesClusterScaleWorker.java	62.50%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #12302   +/-   ##
=========================================
  Coverage     17.58%   17.59%           
- Complexity    15580    15584    +4     
=========================================
  Files          5910     5910           
  Lines        529439   529450   +11     
  Branches      64691    64694    +3     
=========================================
+ Hits          93093    93134   +41     
+ Misses       425861   425827   -34     
- Partials      10485    10489    +4     

Flag	Coverage Δ
uitests	3.57% <ø> (ø)
unittests	18.66% <62.50%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[weizhouapache]

code lgtm

@Pearl1594
can this be extracted to a method and add some unit tests for it ?

[weizhouapache]

thanks @Pearl1594 for the update

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
46.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Pearl1594]

@blueorangutan package

[blueorangutan]

@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 16111

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16116

[DaanHoogland]

@Pearl1594 does it make sense to add an integration test for this? It seems like this is also dependent on how external components work.

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-15037)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55948 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12302-t15037-kvm-ol8.zip
Smoke tests completed. 148 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_scale_kubernetes_cluster	Failure	24.74	test_kubernetes_clusters.py

[kiranchavala]

LGTM

Tested with HA enabled cluster, able to scale up and down

k get nodes
NAME                         STATUS   ROLES           AGE     VERSION
ha-k8s-control-19b6f08ff69   Ready    control-plane   4m34s   v1.33.1
ha-k8s-control-19b6f094e81   Ready    control-plane   3m35s   v1.33.1
ha-k8s-control-19b6f0993fa   Ready    control-plane   3m55s   v1.33.1
ha-k8s-node-19b6f09d994      Ready    <none>          4m4s    v1.33.1
ha-k8s-node-19b6f0a1ccd      Ready    <none>          4m19s   v1.33.1


Scale up

╭─ ~                                                                                                                                        ✔ ╱ kubernetes-admin@kubernetes 󱃾 ╱ 05:11:43 PM
╰─ k get nodes
NAME                         STATUS   ROLES           AGE     VERSION
ha-k8s-control-19b6f08ff69   Ready    control-plane   11m     v1.33.1
ha-k8s-control-19b6f094e81   Ready    control-plane   10m     v1.33.1
ha-k8s-control-19b6f0993fa   Ready    control-plane   11m     v1.33.1
ha-k8s-node-19b6f09d994      Ready    <none>          11m     v1.33.1
ha-k8s-node-19b6f0a1ccd      Ready    <none>          11m     v1.33.1
ha-k8s-node-19b6f108570      Ready    <none>          4m44s   v1.33.1

Scale down 

╭─ ~                                                                                                                                        ✔ ╱ kubernetes-admin@kubernetes 󱃾 ╱ 05:19:03 PM
╰─ k get nodes
NAME                         STATUS   ROLES           AGE   VERSION
ha-k8s-control-19b6f08ff69   Ready    control-plane   13m   v1.33.1
ha-k8s-control-19b6f094e81   Ready    control-plane   12m   v1.33.1
ha-k8s-control-19b6f0993fa   Ready    control-plane   12m   v1.33.1
ha-k8s-node-19b6f09d994      Ready    <none>          12m   v1.33.1
ha-k8s-node-19b6f0a1ccd      Ready    <none>          12m   v1.33.1

2025-12-30 11:49:38,867 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-33:[ctx-a0613513, job-46]) (logid:cfea32e1) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"size\":\"2\",\"serviceofferingid\":\"f95dc588-58c9-4619-ac80-81153daafb81\",\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"0A4lDwM45GIJ0i6p7gPE1f5GdT4\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"247\",\"id\":\"f3b313c6-fba3-4ce1-9e69-c62683dacb24\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"f3b313c6-fba3-4ce1-9e69-c62683dacb24\\\",\\\"interface com.cloud.offering.ServiceOffering\\\":\\\"f95dc588-58c9-4619-ac80-81153daafb81\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"f3b313c6-fba3-4ce1-9e69-c62683dacb24\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":46,"initMsid":32988519728069,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"cfea32e1-6daf-4ef1-aea6-b30966f8fd33"}

2025-12-30 11:49:58,438 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-33:[ctx-a0613513, job-46, ctx-9f0dc9bb]) (logid:cfea32e1) Complete async job-46, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.KubernetesClusterResponse/kubernetescluster/{"id":"f3b313c6-fba3-4ce1-9e69-c62683dacb24","name":"ha-k8s","description":"ha-k8s","zoneid":"2d87a5a6-ec1c-4f6c-9f13-b278214a5ff5","zonename":"ref-trl-10516-k-Mol8-kiran-chavala","serviceofferingid":"f95dc588-58c9-4619-ac80-81153daafb81","serviceofferingname":"CKS","etcdnodes":"0","templateid":"74538369-e569-11f0-b10d-1e00bd0003c5","templatename":"SystemVM Template (KVM)","networkid":"819aa597-06d9