CKS: fix NPE when remove a failed external node #12407

weizhouapache · 2026-01-12T12:20:39Z

Description

This PR fixes #11581

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

github-actions · 2026-01-12T12:22:48Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

weizhouapache · 2026-01-12T12:23:31Z

@blueorangutan package

blueorangutan · 2026-01-12T12:24:05Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

...ernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java

github-actions · 2026-01-12T12:44:24Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

blueorangutan · 2026-01-12T12:52:08Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 16336

codecov · 2026-01-12T12:53:25Z

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 17.59%. Comparing base (2399edd) to head (e166703).
⚠️ Report is 8 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...bernetes/cluster/KubernetesClusterManagerImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #12407   +/-   ##
=========================================
  Coverage     17.59%   17.59%           
+ Complexity    15600    15599    -1     
=========================================
  Files          5910     5910           
  Lines        529733   529755   +22     
  Branches      64719    64724    +5     
=========================================
+ Hits          93218    93227    +9     
- Misses       426023   426035   +12     
- Partials      10492    10493    +1

Flag	Coverage Δ
uitests	`3.60% <ø> (+0.02%)`	⬆️
unittests	`18.66% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

weizhouapache · 2026-01-13T08:03:49Z

@blueorangutan package

blueorangutan · 2026-01-13T08:06:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-01-13T09:21:50Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16345

weizhouapache · 2026-01-13T12:27:51Z

@blueorangutan test

blueorangutan · 2026-01-13T12:30:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-01-14T04:35:03Z

[SF] Trillian test result (tid-15176)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55022 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12407-t15176-kvm-ol8.zip
Smoke tests completed. 148 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	28.67	test_kubernetes_clusters.py

vishesh92

clgtm

vishesh92 · 2026-01-14T10:00:15Z

@blueorangutan package

blueorangutan · 2026-01-14T13:30:07Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-01-14T14:29:18Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16359

weizhouapache · 2026-01-15T09:55:39Z

@blueorangutan test

blueorangutan · 2026-01-15T09:56:33Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-01-15T09:59:08Z

[SF] Trillian Build Failed (tid-15188)

DaanHoogland · 2026-01-15T12:53:14Z

@weizhouapache the issue and PR are marked for 20.3 but the branch is based off of 4.22. What should happen?

kiranchavala

@weizhouapache

Issue is still present

Create a cks cluster
Add a external node to the cks cluster
The external node fails to add due to space issues

Cluster running state
Try to remove the external node

Exception

logs

[root@ref-trl-10752-k-Mol8-kiran-chavala-mgmt1 ~]# cat  /var/log/cloudstack/management/management-server.log  |grep -i "job-47"
2026-01-28 10:27:01,000 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:9e9c5bd4) Add job-47 into job monitoring
2026-01-28 10:27:01,003 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1390913202-17:[ctx-0b3912d3, ctx-14f879bd]) (logid:97df2239) submit async job-47, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"mnK0SnDiHU-7yH8r_GeAGoLWmVk\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"207\",\"id\":\"e2889f42-b5f7-4732-864a-b27c5525c88b\",\"ctxDetails\":\"{\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"9dfc270b-4924-4114-83d2-4cf9117435c5\\\",\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"e2889f42-b5f7-4732-864a-b27c5525c88b\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"e2889f42-b5f7-4732-864a-b27c5525c88b\",\"nodeids\":\"9dfc270b-4924-4114-83d2-4cf9117435c5\",\"cmdEventType\":\"KUBERNETES.CLUSTER.NODES.REMOVE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":47,"initMsid":32985365610879,"instanceId":1,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"dbee196a-4267-4a76-9a50-2eb1cfa82a9b"}
2026-01-28 10:27:01,004 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"mnK0SnDiHU-7yH8r_GeAGoLWmVk\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"207\",\"id\":\"e2889f42-b5f7-4732-864a-b27c5525c88b\",\"ctxDetails\":\"{\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"9dfc270b-4924-4114-83d2-4cf9117435c5\\\",\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"e2889f42-b5f7-4732-864a-b27c5525c88b\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"e2889f42-b5f7-4732-864a-b27c5525c88b\",\"nodeids\":\"9dfc270b-4924-4114-83d2-4cf9117435c5\",\"cmdEventType\":\"KUBERNETES.CLUSTER.NODES.REMOVE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":47,"initMsid":32985365610879,"instanceId":1,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"dbee196a-4267-4a76-9a50-2eb1cfa82a9b"}
2026-01-28 10:27:05,108 ERROR [c.c.k.c.a.KubernetesClusterRemoveWorker] (API-Job-Executor-2:[ctx-7c702c5d, job-47, ctx-a5f8ed50]) (logid:dbee196a) Error trying to remove node 9dfc270b-4924-4114-83d2-4cf9117435c5 from Kubernetes Cluster e2889f42-b5f7-4732-864a-b27c5525c88b: Error during SCP transfer. com.cloud.utils.exception.CloudRuntimeException: Error during SCP transfer.
2026-01-28 10:27:05,121 ERROR [o.a.c.a.c.u.k.c.RemoveNodesFromKubernetesClusterCmd] (API-Job-Executor-2:[ctx-7c702c5d, job-47, ctx-a5f8ed50]) (logid:dbee196a) Failed to remove node(s) from Kubernetes cluster ID: 1 due to: Failed to remove node(s) from Kubernetes cluster ID: 1 org.apache.cloudstack.api.ServerApiException: Failed to remove node(s) from Kubernetes cluster ID: 1
2026-01-28 10:27:05,122 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Complete async job-47, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to remove node(s) from Kubernetes cluster ID: 1 due to: Failed to remove node(s) from Kubernetes cluster ID: 1"}
2026-01-28 10:27:05,123 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Publish async job-47 complete on message bus
2026-01-28 10:27:05,123 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Wake up jobs related to job-47
2026-01-28 10:27:05,123 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Update db status for job-47
2026-01-28 10:27:05,124 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Wake up jobs joined with job-47 and disjoin all subjobs created from job- 47
2026-01-28 10:27:05,128 DEBUG [c.c.a.ApiServer] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Retrieved cmdEventType from job info: KUBERNETES.CLUSTER.NODES.REMOVE
2026-01-28 10:27:05,130 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.RemoveNodesFromKubernetesClusterCmd for job-47
2026-01-28 10:27:05,130 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-2:[ctx-7c702c5d, job-47]) (logid:dbee196a) Remove job-47 from job monitoring

boring-cyborg bot added the component:kubernetes label Jan 12, 2026

weizhouapache force-pushed the 4.22-cks-remove-failed-external-node branch from 9750c2a to 415ebb1 Compare January 12, 2026 12:22

boring-cyborg bot added component:build component:kvm component:primary-storage component:storage component:UI labels Jan 12, 2026

github-actions bot added the status:has-conflicts label Jan 12, 2026

weizhouapache changed the base branch from 4.22 to 4.20 January 12, 2026 12:22

weizhouapache added this to the 4.20.3 milestone Jan 12, 2026

github-actions bot removed the status:has-conflicts label Jan 12, 2026

DaanHoogland linked an issue Jan 12, 2026 that may be closed by this pull request

CKS: NPE when trying to remove a external node from a cks cluster #11581

Open

DaanHoogland reviewed Jan 12, 2026

View reviewed changes

...ernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java Outdated Show resolved Hide resolved

CKS: fix NPE when remove a failed external node

fb18a23

weizhouapache force-pushed the 4.22-cks-remove-failed-external-node branch from 415ebb1 to fb18a23 Compare January 12, 2026 12:44

boring-cyborg bot added component:agent component:integration labels Jan 12, 2026

github-actions bot added the status:has-conflicts label Jan 12, 2026

weizhouapache changed the base branch from 4.20 to 4.22 January 12, 2026 12:44

weizhouapache closed this Jan 12, 2026

weizhouapache reopened this Jan 12, 2026

github-actions bot removed the status:has-conflicts label Jan 12, 2026

Update: still NPE when vm is null as it reads the value at first

e166703

vishesh92 approved these changes Jan 14, 2026

View reviewed changes

DaanHoogland modified the milestones: 4.20.3, 4.22.1 Jan 16, 2026

vladimirpetrov self-assigned this Jan 28, 2026

weizhouapache marked this pull request as ready for review January 28, 2026 09:27

kiranchavala assigned kiranchavala and unassigned vladimirpetrov Jan 28, 2026

kiranchavala reviewed Jan 28, 2026

View reviewed changes

sureshanaparti added this to Apache CloudStack 4.22.1 Feb 3, 2026

sureshanaparti moved this to In Progress in Apache CloudStack 4.22.1 Feb 3, 2026

CKS: fix NPE when remove a failed external node #12407

Are you sure you want to change the base?

CKS: fix NPE when remove a failed external node #12407

Conversation

weizhouapache commented Jan 12, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

weizhouapache commented Jan 12, 2026

Uh oh!

blueorangutan commented Jan 12, 2026

Uh oh!

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

blueorangutan commented Jan 12, 2026

Uh oh!

codecov bot commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache commented Jan 13, 2026

Uh oh!

blueorangutan commented Jan 13, 2026

Uh oh!

blueorangutan commented Jan 13, 2026

Uh oh!

weizhouapache commented Jan 13, 2026

Uh oh!

blueorangutan commented Jan 13, 2026

Uh oh!

blueorangutan commented Jan 14, 2026

Uh oh!

vishesh92 left a comment

Choose a reason for hiding this comment

Uh oh!

vishesh92 commented Jan 14, 2026

Uh oh!

blueorangutan commented Jan 14, 2026

Uh oh!

blueorangutan commented Jan 14, 2026

Uh oh!

weizhouapache commented Jan 15, 2026

Uh oh!

blueorangutan commented Jan 15, 2026

Uh oh!

blueorangutan commented Jan 15, 2026

Uh oh!

DaanHoogland commented Jan 15, 2026

Uh oh!

kiranchavala left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov bot commented Jan 12, 2026 •

edited

Loading

kiranchavala left a comment •

edited

Loading