CI: Velox Staging branch Creation FollowUP by Avinash-Raj · Pull Request #156 · rapidsai/velox-testing

Avinash-Raj · 2025-12-31T09:56:35Z

Summary

This PR enhances the staging branch creation workflows with support for merging additional repositories/branches before applying upstream PRs. This enables integration of external feature branches (e.g., cuDF exchange code) into staging branches alongside selected PRs.

Workflow Order

1. Reset target branch to upstream HEAD
2. Merge additional repository/branch (if specified) → Updates BASE_COMMIT
3. Fetch PR list
4. Test merge compatibility (uses updated BASE_COMMIT)
5. Test pairwise merge compatibility (uses updated BASE_COMMIT)
6. Merge PRs
7. Create staging manifest
8. Push branches

…branch on target repo

ected_file in cudf directory check

simoneves

LGTM

simoneves · 2026-01-23T01:14:17Z

Wait... does this PR have the right root in GH? I thought it was just one commit to update the README file, but it's showing all the original implementation commits too...

paul-aiyedun · 2026-02-03T22:37:30Z

+        type: boolean
+        required: false
+        default: true
+      pr_label:


Nit: pr_labels, since this can be multiple labels.

The same comment applies to other files.

paul-aiyedun · 2026-02-03T22:48:34Z

-    needs: sync-and-aggregate
-    if: inputs.build_and_run_tests != false  # Defaults to true for scheduled runs
-    uses: ./.github/workflows/velox-test.yml
+  velox-create-staging:


Per my previous comment, we should also take in parameters for additional non-PR repos/branches that will be merged in. This will be needed (at least, temporarily) for the cuDF exchange changes.

Added, merge conflict occurs when I tried to pass IBM/velox, ibm-research-preview as additional repo and branch. Relevant CI Run.

paul-aiyedun · 2026-02-03T23:34:53Z

+#!/usr/bin/env bash
+set -euo pipefail
+
+# Presto Staging Branch Creator


I think this verbose comment should be a --help description.

Same comment applies to other files.

paul-aiyedun · 2026-02-03T23:38:38Z

Out of curiosity, did you consider doing this in python? It may make sense for this level of complexity and allow for easier unit testing.

Avinash-Raj · 2026-02-04T10:00:42Z

Successful CI run.

… on velox-create-staging workflow

mattgara

LGTM, just had a few comments related to quality-of-life. We can move any of them to a follow up PR if they would be too complex to implement in this PR.

mattgara · 2026-02-06T07:42:11Z

      auto_fetch_prs:
-        description: Auto-fetch PRs with 'ready-to-merge' label (disable to manually specify PRs)
+        description: >-
+          Auto-fetch non-draft PRs with 'cudf' label


Can this work for a general base repository (for example in the case of Presto?)

We can do this in a follow up PR if not.

Yes, the auto-fetch PR occurs on the repository that is passed as base_repository, not specific to facebookincubator/velox.

mattgara · 2026-02-06T07:44:46Z

+Target path: ${DEFAULT_TARGET_PATH}
+
+Examples:
+  ./presto/scripts/create_staging.sh --manual-pr-numbers "1,2,3"   # Merge specific PRs (required)


Typically we have the convention that we expect to call these scripts from their base folder (as in ./create_staging.sh).

However, this script may have to be called from the base of the git repo to work correctly? If so, please document what the expectation is around where the script is to be called from.

Good catch, but the script could be called from any directory and it should work without any issues. I made examples of calling this script from project root to differentiate between Velox and Presto create_staging.sh scripts.

mattgara · 2026-02-06T07:45:31Z

+
+exec "${PARENT_SCRIPT}" \
+  --target-path "${DEFAULT_TARGET_PATH}" \
+  --base-repository "prestodb/presto" \


hmm can you please explain why this hard-coded call here?

Currently, we are only considering PRs raised against the upstream repositories facebookincubator/velox and presto/prestodb. Guess there might be no change in this unless there is a specific requirement, so I hardcoded the base_repo and base_branch inputs in the sub-scripts.

If we want to add an extra layer on top of base_branch, we can pass the branch and repository where it belongs as additional_repository and additional_branch.

mattgara · 2026-02-06T07:46:21Z

+  [[ -n "${value}" ]] || die "Missing ${name}. Run the appropriate prior step to set it."
+}
+
+retry() {


Hmm can you please explain which commands we expect might fail initially but suceeed on a retry?

All the git fetch commands (helps to grab the PRs from upstream remote) are guarded with retry. This should avoid entire workflow to fail if a single git fetch command failed because of network failure which might likely to occur in a runner machine.

might likely to occur in a runner machine.

Oh interesting, I've never seen a git fetch command fail for me locally, but on a github runner this seems to be a possibility. Good to know, thanks!

mattgara · 2026-02-06T07:49:13Z

+  log "All PRs can merge cleanly with ${BASE_BRANCH}."
+}
+
+test_pairwise_compatibility() {


I believe at one point in this PRs history it would output the conflicts formatted as a table/matrix + a separate table that showed the PR #, link to PR and the PR author.

Could we add this as an optional pretty-print please? This would really help visualize the conflicts almost immediately.

Yes, added now!

mattgara · 2026-02-06T07:50:19Z

+    return 0
+  fi
+
+  maybe_confirm_reset


Can we document clearly what state we expect the local git repo to be when using this tool? For example, can you have dirty/unchecked files (probably a common case.)

Good point, documented this point. It expects the target repo in clean state, if not then it automatically notifies the user about the dirty file and instruct them to stash those before continuing.

Avinash-Raj · 2026-02-06T18:40:05Z

@mattgara I have addressed all of your review comments. Let me know if anything was missed.

paul-aiyedun

I had a few minor comments, but changes overall look good to me.

paul-aiyedun · 2026-02-05T02:54:28Z

+        required: false
+        default: false
+      additional_repository:
+        description: 'Additional repository to merge from (e.g., rapidsai/cudf)'


Remove the rapidsai/cudf reference?

paul-aiyedun · 2026-02-05T14:24:40Z

+
+additional_merge:
+{{ADDITIONAL_MERGE_SECTION}}
+merged_prs:


Nit: Add a new line before this line.

paul-aiyedun · 2026-02-05T14:28:29Z

+BASE_REPO="facebookincubator/velox"
+BASE_BRANCH="main"
+TARGET_REPO=""
+TARGET_BRANCH="staging"
+TARGET_PATH=""
+WORK_DIR="velox"


If we intend on using this script for both Velox and Presto, then we should not have Velox default values.

Agreed, edited!

paul-aiyedun · 2026-02-06T19:12:08Z

+      additional_repository:
+        description: 'Additional repository to merge from (e.g., IBM/velox for cuDF exchange)'
+        type: string
+        required: false
+        default: 'rapidsai/velox'
+      additional_branch:
+        description: 'Branch from additional repository to merge (e.g., ibm-research-preview for cuDF exchange)'


Remove IBM references.

mattgara

LGTM, thanks

Avinash-Raj · 2026-02-07T03:06:41Z

@paul-aiyedun Thanks for your review, I have addressed all of your comments!

Avinash-Raj added 22 commits December 31, 2025 09:53

repo name fix

d17e6f7

altered the bash increment

bb50b41

Added readme file for CI workflows

684efe2

Added code to optionally create a PR from staging branch to the main …

dcbc497

…branch on target repo

removed PR aff

c5e06a9

ected_file in cudf directory check

removed pR creation step

7874c1a

exit the workflow if no PRs found

1106429

typo fixed

e2f0c2f

remove ready-to-merge label and consider only non-draft PRs

aa27cfc

Add steps to show merge conflict with main and pair wise merge conflicts

6f82eb3

Merge branch 'main' into ci-staging-branch-creation-followup

03ca362

Merge branch 'main' into ci-staging-branch-creation-followup

a1305d2

removed PAT token refernce

af969f2

typo

4fe6972

Added legend table and modified pairwise PR check table to look nice

79df192

Merge branch 'main' into ci-staging-branch-creation-followup

d2f8bea

properly align pair-wise PR matrix table

e2c50ff

typo

5ccfbf3

Added manifest file and push to staging and staging-date branches

96f0879

fixed velox testing path

244ab1c

added secrets.GH_TOKEN as env var to branch push step

f2dc999

Added permissions

43b743b

Avinash-Raj requested review from mattgara, misiugodfrey, paul-aiyedun and simoneves January 22, 2026 18:54

Upadted staging ci on workflow readme

84a9c68

simoneves approved these changes Jan 23, 2026

View reviewed changes

Use VELOX_FORK_PAT secret for passing the target repo PAT token

3a54267

Avinash-Raj added 6 commits January 30, 2026 06:13

typo

b99d91a

pass uppercase vars to emit_output

b074e77

register presto create staging workflow

7012648

typo

d3ab004

use velox_fork_pat secret for presto staging ci

19eb821

Updated readme

10fb835

paul-aiyedun reviewed Feb 3, 2026

View reviewed changes

Avinash-Raj added 2 commits February 4, 2026 06:32

Renamed pr_label to pr_labels

4bdd461

Added support for merging from additional repos branch

8d758fc

Avinash-Raj added 7 commits February 4, 2026 10:30

Updated create_staging.sh --help

38e5759

Merge branch 'main' into ci-staging-branch-creation-followup

d603fa6

move merge additional branch step just before to fetching PR list step

4bb1a92

specify additional branch in manifest file

8e07fc1

set default values for additional_repo and addition_branch GHA inputs…

48a48e0

… on velox-create-staging workflow

Updated workflow readme w.r.t additional repo and branch

dae3655

Order change in standalone script

634c5d4

Avinash-Raj requested a review from paul-aiyedun February 5, 2026 17:18

mattgara reviewed Feb 6, 2026

View reviewed changes

Avinash-Raj added 2 commits February 6, 2026 18:12

create_staging.sh updated to show Pairwise PR matrix table

f8e2f65

--exclude-pr-numbers cli flag

8a77c23

Avinash-Raj requested a review from mattgara February 6, 2026 18:38

notify users about the dirty state, documented

a0df72a

paul-aiyedun approved these changes Feb 6, 2026

View reviewed changes

mattgara approved these changes Feb 7, 2026

View reviewed changes

Avinash-Raj merged commit f8e631d into main Feb 7, 2026
3 of 4 checks passed

Avinash-Raj deleted the ci-staging-branch-creation-followup branch February 7, 2026 03:07

Final cleanup

9af4736

Conversation

Avinash-Raj commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Workflow Order

Uh oh!

simoneves left a comment

Choose a reason for hiding this comment

Uh oh!

simoneves commented Jan 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Avinash-Raj commented Feb 4, 2026

Uh oh!

mattgara left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Avinash-Raj Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattgara Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Avinash-Raj commented Feb 6, 2026

Uh oh!

paul-aiyedun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattgara left a comment

Avinash-Raj commented Dec 31, 2025 •

edited

Loading

Avinash-Raj Feb 6, 2026 •

edited

Loading

mattgara Feb 6, 2026 •

edited

Loading