Update etcd to 3.6 for Kubernetes 1.34 #12634

tmurakam · 2025-10-17T08:45:39Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
Update etcd from 3.5.x to 3.6.6

Disable backing up etcd v2 data for etcd >=3.6.0
Use etcd 3.5 for netchecker
For upgrading existing etcd 3.5 to 3.6:
- Add version check of current etcd before upgrade to 3.6 (It must be >=3.5.20). Please see following URLs:
  - https://etcd.io/blog/2025/upgrade_from_3.5_to_3.6_issue/
  - https://etcd.io/blog/2025/upgrade_from_3.5_to_3.6_issue_followup/
- Cleanup v2 store by setting ETCD_SNAPSHOTS_COUNT=1 and restart etcd.

See also #12629

Which issue(s) this PR fixes:
Fixes #12629

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Update etcd to 3.6.6

k8s-ci-robot · 2025-10-17T08:45:42Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-10-17T08:45:46Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tmurakam
Once this PR has been reviewed and has the lgtm label, please assign tico88612 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tmurakam · 2025-10-17T08:46:45Z

/ok-to-test

tmurakam · 2025-10-17T08:47:57Z

/hold

tmurakam · 2025-10-20T01:16:36Z

/unhold

tico88612 · 2025-10-23T16:41:01Z

@tmurakam Do you think we need to be aware of this? Upgrade to 3.5.24 first?
https://etcd.io/blog/2025/upgrade_from_3.5_to_3.6_issue_followup/

tmurakam · 2025-10-23T23:46:15Z

@tico88612
It recommended to upgrade 3.5.24, but there is a workaround restarting etcd (3.5.20-3.5.23) before upgrading.
I already put a restart task before upgrading, so it is not mandatory.

EDIT: Note: The etcd version of the Kubespray 2.29.0 is 3.5.23.

tmurakam · 2025-10-31T02:39:00Z

@tico88612
I believe all corrections have been made. Would you please review it?

tico88612 · 2025-11-06T03:59:37Z

/cc

tico88612

Some comments

tico88612 · 2025-11-09T14:35:04Z

roles/etcd/tasks/clean_v2_store.yml

+- name: Debug etcd_current_version
+  ansible.builtin.debug:
+    msg: "{{ etcd_current_version }}"
+  when: etcd_cluster_setup


Should we need this debug message?

Not mandatory.
But this is the last chance to show the etcd version before installing a new one, so this might be useful for troubleshooting.

roles/etcd/templates/etcd.env.j2

VannTen · 2025-11-14T08:24:42Z

IMO, I'd first do a simple upgrade to 3.5.24 for one release, backport it to 2.29 (and previous branch as well) to be a little extra careful.

Not sure if we should delay the upgrade to 3.6 until after 2.30 though 🤔

linux-foundation-easycla · 2025-11-18T13:00:17Z

The committers listed above are authorized under a signed CLA.

✅ login: tmurakam / name: Takuya Murakami (8b753a1, b4bd243)

tmurakam · 2025-11-19T15:45:15Z

/retest

VannTen · 2025-12-04T14:13:06Z

roles/etcd/tasks/clean_v2_store.yml

+      when:
+        - etcd_current_version is version('3.5.20', '<')
+      fail:
+        msg: "You need to upgrade etcd to 3.5.20 or later before upgrade to 3.6. Current version is {{ etcd_current_version }}."


I think now that release-2.29 has etcd 3.5.25, we should make a patch release for 2.29 ASAP, and require etcd to be at least 3.5.24 for the upgrade. Once we have the release we can probably also mention the kubespray version to be extra helpful, IMO.

I will do this release cut in next week.

Ok, I will change this version check to >=3.5.24

- Add hashes for etcd 3.6.6 - Remove etcd v2 backup task for etcd 3.6 The etcd 3.6 removes 'etcdctl backup' command with ETCDCTL_API=2 - Downgrade etcd to 3.5 in netchecker The netchecker does not work with etcd 3.6 becaust it removes v2 API support (--enable-v2). And netchekcer does not support v3 API.

… 3.6

k8s-ci-robot requested review from ErikJiang and mzaian October 17, 2025 08:45

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 17, 2025

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Oct 17, 2025

tmurakam marked this pull request as ready for review October 17, 2025 08:47

k8s-ci-robot requested review from cyclinder and yankay October 17, 2025 08:47

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 17, 2025

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from 230570c to 322a58b Compare October 17, 2025 09:25

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 17, 2025

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from 322a58b to 57122c3 Compare October 17, 2025 11:24

tmurakam changed the title ~~[WIP] Update etcd to 3.6 for Kubernetes 1.34~~ Update etcd to 3.6 for Kubernetes 1.34 Oct 20, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 20, 2025

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 20, 2025

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from 57122c3 to 7d8e797 Compare October 23, 2025 23:54

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 23, 2025

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from 7d8e797 to ed51a39 Compare October 23, 2025 23:57

k8s-ci-robot requested a review from tico88612 November 6, 2025 03:59

tico88612 reviewed Nov 9, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 14, 2025

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from 8b5746d to aceeabc Compare November 15, 2025 06:49

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from f528f20 to 5997631 Compare November 15, 2025 06:57

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed do-not-merge/contains-merge-commits size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 15, 2025

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch 3 times, most recently from d1aac94 to 2723b97 Compare November 18, 2025 13:00

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 18, 2025

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from 2723b97 to ea58fe4 Compare November 18, 2025 13:02

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Nov 18, 2025

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from ea58fe4 to c478640 Compare December 1, 2025 00:38

VannTen reviewed Dec 4, 2025

View reviewed changes

tmurakam added 2 commits December 6, 2025 16:28

Fix: Change etcd config to clean up v2 store before upgrading etcd to…

8b753a1

… 3.6

tmurakam force-pushed the feature/kubernetes-1.34-etcd-3.6 branch from c478640 to 8b753a1 Compare December 6, 2025 07:30

Update etcd to 3.6 for Kubernetes 1.34 #12634

Are you sure you want to change the base?

Update etcd to 3.6 for Kubernetes 1.34 #12634

Conversation

tmurakam commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Oct 17, 2025

Uh oh!

k8s-ci-robot commented Oct 17, 2025

Uh oh!

tmurakam commented Oct 17, 2025

Uh oh!

tmurakam commented Oct 17, 2025

Uh oh!

tmurakam commented Oct 20, 2025

Uh oh!

tico88612 commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmurakam commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmurakam commented Oct 31, 2025

Uh oh!

tico88612 commented Nov 6, 2025

Uh oh!

tico88612 left a comment

Choose a reason for hiding this comment

Uh oh!

tico88612 Nov 9, 2025

Choose a reason for hiding this comment

Uh oh!

tmurakam Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

VannTen commented Nov 14, 2025

Uh oh!

linux-foundation-easycla bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmurakam commented Nov 19, 2025

Uh oh!

VannTen Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tico88612 Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

tmurakam Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tmurakam commented Oct 17, 2025 •

edited

Loading

tico88612 commented Oct 23, 2025 •

edited

Loading

tmurakam commented Oct 23, 2025 •

edited

Loading

linux-foundation-easycla bot commented Nov 18, 2025 •

edited

Loading

VannTen Dec 4, 2025 •

edited

Loading