feat(ci): automate release process by Krishna-kg732 · Pull Request #3536 · kubeflow/trainer

Krishna-kg732 · 2026-05-20T17:33:18Z

Summary

This PR continues the work started in #3148 by @milinddethe15.

Changes

Automated release process with changelog generation using git-cliff
Split monolithic CHANGELOG.md into per-release-line files:
- CHANGELOG/CHANGELOG-1.x.md
- CHANGELOG/CHANGELOG-2.0.md
- CHANGELOG/CHANGELOG-2.1.md
- CHANGELOG/CHANGELOG-2.2.md
Updated release workflows with manual trigger and image build steps
Refactored GitHub release job to integrate changelog generation
Fixed release branch naming to use major.minor version format

Testing

Tested PyPI publishing workflow on fork
Verified changelog generation and split across release lines

Copilot

Pull request overview

This PR introduces a more automated release process for Kubeflow Trainer, centered around git-cliff changelog generation, new CI workflows for release validation/release publishing, and a move from a single monolithic changelog to per-minor-series changelog files.

Changes:

Add make release + hack/release.sh to prepare release commits and generate per-minor changelog updates via git-cliff.
Add/refresh GitHub Actions workflows for release validation (check-release.yaml) and release automation (release.yaml), plus manual dispatch support for image/chart publishing.
Split CHANGELOG.md into per-release-line markdown files under CHANGELOG/.

Reviewed changes

Copilot reviewed 17 out of 19 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
README.md	Updates changelog link to point at the new `CHANGELOG/` structure.
Makefile	Adds a `release` target that invokes `hack/release.sh`.
hack/release.sh	New script to bump versions/tags in repo files and generate changelog content via `git-cliff`.
docs/release/README.md	Updates release documentation to include the new `make release` workflow and changelog location.
docs/release/changelog.py	Removes the old PyGithub-based changelog generator script.
cliff.toml	Adds `git-cliff` configuration used by release tooling/workflows.
CHANGELOG/README.md	Adds an index for per-minor changelog files.
CHANGELOG/CHANGELOG-*.md	Adds split changelog files for 1.x, 2.0, 2.1, 2.2 lines.
api/python_api/pyproject.toml	Ensures wheel build packages the `kubeflow_trainer_api` package.
.github/workflows/template-publish-image/action.yaml	Adds a manual-dispatch tag metadata tag rule for image publishing.
.github/workflows/release.yaml	New release automation workflow (build/publish Python, tag, GitHub release, trigger images/charts).
.github/workflows/check-release.yaml	New PR-time validation for release PRs (VERSION/tag/manifests/chart/python version checks).
.github/workflows/publish-helm-charts.yaml	Adds `workflow_dispatch` support + concurrency settings.
.github/workflows/build-and-push-images.yaml	Adds `workflow_dispatch` and allows publishing on manual dispatch.
.github/workflows/check-pr-title.yaml	Adds `area/release` label to PR-title check ignore list.

Comments suppressed due to low confidence (2)

hack/release.sh:139

This script prints guidance to open a PR against main, but the repo workflows and docs use master as the default branch; update the message to avoid sending releasers to the wrong base branch.

echo "  2. Open a PR:"
echo "     - For latest minor series: open a PR to 'main' and get it reviewed and merged"
echo "     - For older minor series patch (e.g. ${MAJOR_VERSION}.${MINOR_VERSION}.Z when main"
echo "       is at a newer minor): checkout the release-${MAJOR_VERSION}.${MINOR_VERSION} branch,"
echo "       commit changes, and open a PR to release-${MAJOR_VERSION}.${MINOR_VERSION}"

docs/release/README.md:20

The release guide still instructs installing PyGithub and later running python docs/release/changelog.py, but this PR deletes docs/release/changelog.py and switches to git-cliff; update the prerequisites/instructions to remove the PyGithub dependency and reference the new make release/git-cliff flow consistently.

- Create a [GitHub Token](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token).

- Install `PyGithub` to generate the [Changelog](./../../CHANGELOG/README.md):

pip install PyGithub>=1.55

andreyvelich

Thanks @Krishna-kg732 for picking up this work — automating the release flow is a meaningful improvement. This review was conducted with AI tooling assistance; inline comments highlight specific items to address.

google-oss-prow · 2026-05-22T18:14:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from andreyvelich. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich · 2026-05-25T13:15:56Z

@Krishna-kg732 Did you get a chance to fix the remaining comments ?

Krishna-kg732 · 2026-05-25T18:42:36Z

Hey @andreyvelich , yup , I will test with new the changes and then I think we will be ready to merge

…n checks Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): Implement automated release process with changelog generation Signed-off-by: milinddethe15 <milinddethe15@gmail.com> fix lint Signed-off-by: milinddethe15 <milinddethe15@gmail.com> enhance tag validation and branch creation Signed-off-by: milinddethe15 <milinddethe15@gmail.com> fix: Add 'area/release' to ignored labels in PR title check Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): Enhance release workflows with manual trigger and image build steps Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): Add PyPI API token for publishing packages Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): Update concurrency settings in publish-helm-charts workflow and upgrade git-cliff-action version Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): Update git-cliff-action args to include --unreleased option Signed-off-by: milinddethe15 <milinddethe15@gmail.com> refactor: reorganize release workflow to generate changelog before GitHub release Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): Update GitHub release action to remove changelog prepending and simplify release name Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): Refactor GitHub release job to integrate changelog generation and simplify workflow Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): create_branch_and_tag job to create_branch and streamline tagging process Signed-off-by: milinddethe15 <milinddethe15@gmail.com> rebase commit Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com> fix: update release branch naming to use major.minor version format Signed-off-by: milinddethe15 <milinddethe15@gmail.com> fix endline Signed-off-by: milinddethe15 <milinddethe15@gmail.com> feat(release): update README with upstream tag fetching instructions Signed-off-by: milinddethe15 <milinddethe15@gmail.com> Release v2.2.2 Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com> Revert "Release v2.2.2" chore: configure TestPyPI publishing for fork testing feat: split changelog by release line Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com> chore:removed the monoloith CHANGELOG.md Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com> chore: capitalize Changelog file name Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com> chore: revert fork testing config Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com> chore:latest minor release support updated Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com> chore:resolve changelog casing conflict and restore correct file Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com> chore: resolve git casing conflict Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com> feat: split changelog by release line Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com> chore: capitalize Changelog file name Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com> chore:latest minor release support updated Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com> chore: remove duplicate lowercase changelog file Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com>

Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

…sset updates to release workflow Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

andreyvelich · 2026-06-02T17:13:29Z

+      - name: Generate changelog
+        id: changelog
+        uses: orhun/git-cliff-action@v4
+        with:
+          config: cliff.toml
+          args: >
+            --unreleased
+            --tag ${{ needs.prepare.outputs.tag }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}


Generation of changelog should be part of make release command since it might require some manual adjustment, similar to how we do in SDK, only extract changelog for publish:

https://github.com/kubeflow/sdk/blob/main/Makefile#L90-L94

https://github.com/kubeflow/sdk/blob/main/.github/workflows/release.yml#L177

cc @Fiona-Waters @kramaranya

andreyvelich · 2026-06-02T17:13:55Z


    env:
-      SHOULD_PUBLISH: ${{ github.event_name == 'push' }}
+      SHOULD_PUBLISH: ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' }}


Why do you need this?

andreyvelich · 2026-06-02T17:15:47Z

+      - name: Publish release artifacts
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const tag = '${{ needs.prepare.outputs.tag }}';
+            const workflows = [
+              'build-and-push-images.yaml',
+              'publish-helm-charts.yaml',
+            ];
+            const timeoutMs = 90 * 60 * 1000;
+            const intervalMs = 60 * 1000;
+            const deadline = Date.now() + timeoutMs;
+
+            const dispatchedRuns = await Promise.all(workflows.map(async (workflow_id) => {
+              const response = await github.rest.actions.createWorkflowDispatch({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                workflow_id,
+                ref: tag,
+                return_run_details: true,
+              });
+              const runId = response.data.workflow_run_id;
+              if (!runId) {
+                throw new Error(`GitHub did not return a run ID for ${workflow_id}`);
+              }
+              core.info(`Dispatched ${workflow_id}: ${response.data.html_url}`);
+              return { workflow_id, runId };
+            }));
+
+            for (const { workflow_id, runId } of dispatchedRuns) {
+              core.info(`Waiting for ${workflow_id} to complete for ${tag}`);
+              let run;
+
+              while (Date.now() < deadline) {
+                const response = await github.rest.actions.getWorkflowRun({
+                  owner: context.repo.owner,
+                  repo: context.repo.repo,
+                  run_id: runId,
+                });
+                run = response.data;
+
+                if (run?.status === 'completed') {
+                  if (run.conclusion === 'success') {
+                    core.info(`${workflow_id} completed successfully: ${run.html_url}`);
+                    break;
+                  }
+                  core.setFailed(`${workflow_id} concluded with ${run.conclusion}: ${run.html_url}`);
+                  return;
+                }
+
+                await new Promise((resolve) => setTimeout(resolve, intervalMs));
+              }
+
+              if (run?.status !== 'completed') {
+                core.setFailed(`Timed out waiting for ${workflow_id} to complete for ${tag}`);
+                return;
+              }


Why do you need all of these?
We don't do this in SDK: https://github.com/kubeflow/sdk/blob/main/.github/workflows/release.yml#L177

andreyvelich · 2026-06-02T17:18:06Z

+{%- for group_name in group_order %}
+{%- set group_commits = commits | filter(attribute="group", value=group_name) -%}
+{%- if group_commits | length > 0 %}
+### {{ group_name }}
+
+{% for commit in group_commits | reverse -%}
+{%- set message = commit.message | split(pat="\n") | first | trim -%}
+{%- set parts = message | split(pat=" (#") -%}
+{%- set author = commit.remote.username | default(value=commit.author.name) -%}
+   {% if parts | length > 1 and parts | last | trim | split(pat=")") | length > 1 -%}
+      {%- set pr_part = parts | last | trim -%}
+      {%- set pr_number = pr_part | replace(from=")", to="") -%}
+      - {{ parts | slice(end=-1) | join(sep=" (#") }} ([#{{ pr_number }}](https://github.com/kubeflow/trainer/pull/{{ pr_number }}) by @{{ author }})
+   {% else -%}
+      - {{ message }} (@{{ author }})
+   {% endif -%}
+{% endfor %}
+
+{%- endif %}
+{%- endfor %}
+
+{%- if github -%}
+{%- set new_contributors = github.contributors | filter(attribute="is_first_time", value=true) -%}
+{%- if new_contributors | length != 0 %}
+
+### New Contributors
+{%- for contributor in new_contributors %}
+* @{{ contributor.username }} made their first contribution in \
+  [#{{ contributor.pr_number }}](https://github.com/kubeflow/trainer/pull/{{ contributor.pr_number }})
+{%- endfor %}
+{%- endif %}
+{%- endif -%}


Could you follow the format like in Kubeflow SDK ?

https://github.com/kubeflow/sdk/blob/main/cliff.toml#L24C1-L48

Copilot AI review requested due to automatic review settings May 20, 2026 17:33

google-oss-prow Bot requested review from akshaychitneni and jinchihe May 20, 2026 17:33

google-oss-prow Bot added the size/XXL label May 20, 2026

Copilot started reviewing on behalf of Krishna-kg732 May 20, 2026 17:33 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

andreyvelich requested changes May 22, 2026

View reviewed changes

google-oss-prow Bot assigned andreyvelich May 22, 2026

andreyvelich mentioned this pull request May 22, 2026

Update k8s 1 36 kubernetes-sigs/kueue#10936

Merged

andreyvelich reviewed May 26, 2026

View reviewed changes

milinddethe15 and others added 7 commits June 2, 2026 22:30

Chore:remove the Test release changeLog update

95202a5

Signed-off-by: krishna-kg732 <krishnagupta.kg2k6@gmail.com>

Potential fix for pull request finding (fix by copilot)

cf3af2f

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

chore:address review comments

055b6d1

Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

chore:run pip-compile

7694616

Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

chore: address review

d29c264

Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

fix(release): wait for artifact publishing before GitHub release

bb297c6

Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

Krishna-kg732 force-pushed the feat/automate-release branch 2 times, most recently from e59f436 to 0df9a5a Compare June 2, 2026 17:10

refactor(release): delete release.sh, inline make release, and move a…

6b71b68

…sset updates to release workflow Signed-off-by: Krishna Gupta <Krishnagupta.kg2k6@gmail.com>

Krishna-kg732 force-pushed the feat/automate-release branch from 0df9a5a to 6b71b68 Compare June 2, 2026 17:12

andreyvelich reviewed Jun 2, 2026

View reviewed changes

Conversation

Krishna-kg732 commented May 20, 2026

Summary

Changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

google-oss-prow Bot commented May 22, 2026

Uh oh!

andreyvelich commented May 25, 2026

Uh oh!

Krishna-kg732 commented May 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andreyvelich Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

andreyvelich Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

andreyvelich Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

andreyvelich Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

andreyvelich Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants