feat: add stopping stage update run implementation #374

britaniar · 2025-12-10T03:46:39Z

Description of your changes

I have:

Added a check to make sure no cluster start updating and let currently updating cluster finish within a stage before marking the stage and update run as stopped.
Update the stage condition status as stopping or stopped.
Update integration tests and added UTs.
Run make reviewable to ensure this PR is ready for review.

How has this code been tested

Integration Test
Unit Test

Special notes for your reviewer

While waiting for cluster to finish updating, stage and update run will have a progressing unknown condition with the reason as stopping.

codecov · 2025-12-10T18:10:38Z

Codecov Report

❌ Patch coverage is 87.41722% with 19 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/controllers/updaterun/stop.go	88.37%	11 Missing and 4 partials ⚠️
pkg/controllers/updaterun/controller.go	76.47%	2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

pkg/controllers/updaterun/controller.go

pkg/controllers/updaterun/execution.go

pkg/controllers/updaterun/stop.go

pkg/controllers/updaterun/execution.go

pkg/controllers/updaterun/stop.go

apis/placement/v1beta1/stageupdate_types.go

pkg/controllers/updaterun/stop.go

pkg/controllers/updaterun/execution_integration_test.go

pkg/controllers/updaterun/stop.go

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

pkg/controllers/updaterun/stop.go

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

pkg/controllers/updaterun/stop.go

pkg/controllers/updaterun/stop_test.go

pkg/controllers/updaterun/stop_integration_test.go

Arvindthiru · 2025-12-16T20:03:05Z

pkg/controllers/updaterun/stop_integration_test.go

+			validateUpdateRunMetricsEmitted(wantMetrics...)
+		})
+
+		It("Should delete all the clusterResourceBindings in the delete stage and complete the update run", func() {


Can we add a case where we stop in between the deletion stage ?

I tried previously and we have no control over the clusters in the stage so the stop timing here is difficult.

We can add a finalizer on the bindings so that they are not deleted.

pkg/controllers/updaterun/stop_test.go

pkg/controllers/updaterun/stop.go

pkg/controllers/updaterun/stop_test.go

pkg/controllers/updaterun/stop_integration_test.go

jwtty · 2025-12-17T00:01:31Z

pkg/controllers/updaterun/stop_integration_test.go

+			validateUpdateRunMetricsEmitted(wantMetrics...)
+		})
+
+		It("Should wait for cluster to finish updating before update run is completely stopped", func() {


This "It" section is redundant unless you want to check updateRun is in stopping state CONSISTENTLY.

pkg/controllers/updaterun/stop_integration_test.go

jwtty · 2025-12-17T00:16:17Z

pkg/controllers/updaterun/stop_integration_test.go

+			validateUpdateRunMetricsEmitted(wantMetrics...)
+		})
+
+		It("Should delete all the clusterResourceBindings in the delete stage and complete the update run", func() {


We can add a finalizer on the bindings so that they are not deleted.

jwtty · 2025-12-17T00:21:35Z

pkg/controllers/updaterun/stop_integration_test.go

+	"github.com/kubefleet-dev/kubefleet/pkg/utils/condition"
+)
+
+var _ = Describe("UpdateRun stop tests", func() {


Not a blocker to this PR, but this test is too complicated, including many unnecessary tests for stopping. If we can create a small number of updating/deleting clusters and a simpler strategy to just test the stop cases, that's better. We actually have one https://github.com/kubefleet-dev/kubefleet/blob/main/pkg/controllers/updaterun/controller_integration_test.go#L473C1-L490C2 This one does not have any to-be-deleted bindings though. Please feel free to add if you want to update.

britaniar · 2025-12-17T19:50:28Z

Will address any unresolved stop integration test specific comments in a follow up PR. Current integration test and UTs should have major coverage.
In the follow up PR I will reduce the integration so that we cover beforeStage -> stop -> cluster 1 -> stop -> afterStage -> stop -> deleteStage -> stop -> continue to complete.

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

Arvindthiru · 2025-12-17T23:49:48Z

pkg/controllers/updaterun/stop.go

+		}
+	}
+
+	if allDeletingClustersDeleted || len(toBeDeletedBindings) == 0 {


Do we need to check if len(toBeDeletedBindings), if len(toBeDeletedBindings) == 0 wouldn't allDeletingClustersDeleted be true ?

Arvindthiru · 2025-12-17T23:55:12Z

pkg/controllers/updaterun/stop_test.go

+				tt.updateRun.Status.StagesStatus[0].Conditions,
+				string(placementv1beta1.StageUpdatingConditionProgressing),
+			)
+			if progressingCond == nil {


cmp.Diff can handle nil values passed to it

Arvindthiru · 2025-12-18T00:19:09Z

pkg/controllers/updaterun/stop_test.go

+							},
+						},
+					},
+					UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{


nit: do we need UpdateStrategySnapshot for this UT ?

Arvindthiru · 2025-12-18T00:49:38Z

pkg/controllers/updaterun/stop_test.go

+			}
+
+			// Check error expectations.
+			if tt.wantError {


Lets check the error similar to TestStopUpdatingStage, i.e. check the error msg instead of checking to see if error is errStagedUpdatedAborted, if we construct the wantErr correctly we don't need to use errors.Is ?

britaniar force-pushed the stopUpdateRun branch from 040bf28 to 878fa65 Compare December 10, 2025 17:34

britaniar force-pushed the stopUpdateRun branch from 878fa65 to 723e9c9 Compare December 11, 2025 02:06

britaniar added 2 commits December 10, 2025 20:17

stopping update run implementation

220e29f

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

fix integration test

4bee175

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

britaniar force-pushed the stopUpdateRun branch from 723e9c9 to 4bee175 Compare December 11, 2025 17:23

britaniar marked this pull request as ready for review December 11, 2025 17:23

add UT coverage

3e3eb79

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

Arvindthiru reviewed Dec 11, 2025

View reviewed changes

pkg/controllers/updaterun/controller.go Show resolved Hide resolved

Arvindthiru reviewed Dec 11, 2025

View reviewed changes

pkg/controllers/updaterun/controller.go Show resolved Hide resolved

Arvindthiru reviewed Dec 11, 2025

View reviewed changes

pkg/controllers/updaterun/execution.go Outdated Show resolved Hide resolved

Arvindthiru reviewed Dec 12, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Outdated Show resolved Hide resolved

Arvindthiru reviewed Dec 12, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Outdated Show resolved Hide resolved

Arvindthiru reviewed Dec 12, 2025

View reviewed changes

pkg/controllers/updaterun/execution.go Outdated Show resolved Hide resolved

Arvindthiru reviewed Dec 12, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Outdated Show resolved Hide resolved

jwtty reviewed Dec 12, 2025

View reviewed changes

address comments

9d05726

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

Arvindthiru reviewed Dec 12, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Show resolved Hide resolved

Arvindthiru reviewed Dec 12, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Outdated Show resolved Hide resolved

Arvindthiru reviewed Dec 12, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Outdated Show resolved Hide resolved

address minor comments

77a73ab

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

Arvindthiru reviewed Dec 16, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Outdated Show resolved Hide resolved

Arvindthiru reviewed Dec 16, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Show resolved Hide resolved

Arvindthiru reviewed Dec 16, 2025

View reviewed changes

pkg/controllers/updaterun/stop.go Show resolved Hide resolved

Arvindthiru reviewed Dec 16, 2025

View reviewed changes

pkg/controllers/updaterun/stop_test.go Show resolved Hide resolved

Arvindthiru reviewed Dec 16, 2025

View reviewed changes

pkg/controllers/updaterun/stop_integration_test.go Outdated Show resolved Hide resolved

Arvindthiru reviewed Dec 16, 2025

View reviewed changes

pkg/controllers/updaterun/stop_test.go Show resolved Hide resolved

jwtty reviewed Dec 17, 2025

View reviewed changes

address most comments

eb10bb3

Signed-off-by: Britania Rodriguez Reyes <britaniar@microsoft.com>

britaniar force-pushed the stopUpdateRun branch from 0ac9549 to eb10bb3 Compare December 17, 2025 21:17

Arvindthiru reviewed Dec 17, 2025

View reviewed changes

Arvindthiru reviewed Dec 18, 2025

View reviewed changes

feat: add stopping stage update run implementation #374

Are you sure you want to change the base?

feat: add stopping stage update run implementation #374

Uh oh!

Conversation

britaniar commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of your changes

How has this code been tested

Special notes for your reviewer

Uh oh!

codecov bot commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

britaniar commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Arvindthiru Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

britaniar commented Dec 10, 2025 •

edited

Loading

codecov bot commented Dec 10, 2025 •

edited

Loading

britaniar commented Dec 17, 2025 •

edited

Loading

Arvindthiru Dec 18, 2025 •

edited

Loading