Generate a manifest file for Pytorch builds by erman-gurses · Pull Request #2547 · ROCm/TheRock

erman-gurses · 2025-12-15T07:09:47Z

Closes #2481
Generate a manifest file for Pytorch builds.

Example: therock-manifest_torch_py3.11_release-2.7.json

{
  "pytorch": {
    "commit": "01078254ea873c32a090304cecc6a70c1c87132c",
    "repo": "https://github.com/ROCm/pytorch.git"
  },
  "pytorch_audio": {
    "commit": "95c61b4168fc5133be8dd8c1337d929d066ae6cf",
    "repo": "https://github.com/pytorch/audio"
  },
  "pytorch_vision": {
    "commit": "59a3e1f9f78cfe44cb989877cc6f4ea77c8a75ca",
    "repo": "https://github.com/pytorch/vision"
  },
  "triton": {
    "commit": "b2eaccd6b2b7909bf02ac7cb2dddfa87275c3f25",
    "repo": "https://github.com/ROCm/triton.git"
  },
  "therock": {
    "commit": "905eb116096064c5058b6c487239411637bab52e",
    "repo": "https://github.com/ROCm/TheRock.git",
    "branch": "users/erman-gurses/add-pytorch-manifest"
  }
}

https://therock-dev-artifacts.s3.us-east-2.amazonaws.com/21179896705-linux/manifests/gfx110X-all/therock-manifest_torch_py3.11_release-2.7.json

erman-gurses · 2026-01-05T16:50:31Z

Generated Manifest file:
Linux: https://therock-dev-artifacts.s3.us-east-2.amazonaws.com/external-builds/pytorch/20707403295/gfx94X-dcgpu/therock_manifest-pytorch-wheels-linux-x86_64-20707403295.json

erman-gurses · 2026-01-05T21:38:48Z

Generated Manifest file:
Windows https://therock-dev-artifacts.s3.us-east-2.amazonaws.com/external-builds/pytorch/20725655858/gfx1151/therock_manifest-pytorch-wheels-windows-amd64-20725655858.json

marbre

Some additional feedback but need to take a closer look.

marbre · 2026-01-05T22:08:27Z

+        if: ${{ github.repository_owner == 'ROCm' }}
+        run: |
+          aws s3 cp ${{ env.PACKAGE_DIST_DIR }}/manifests/ \
+            s3://${{ env.S3_BUCKET_ARTIFACTS }}/external-builds/pytorch/${{ github.run_id }}/${{ inputs.amdgpu_family }}/ \


@ScottTodd any feelings about the location?

Let's think carefully about the top level folder structure.

Here's an example Linux release package build: https://github.com/ROCm/TheRock/actions/runs/20737916627#summary-59538918041

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/logs/gfx94X-dcgpu/index.html

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/logs/gfx94X-dcgpu/build_time_analysis.html

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/index-gfx94X-dcgpu.html

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/manifests/gfx94X-dcgpu/therock_manifest.json

Could we match that and write to ${{ github.run_id }}-{{ platform }}/manifests/{{ inputs.amdgpu_family }}/therock_torch_manifest.json?

For example:

20737916627-linux/manifests/gfx94X-dcgpu/therock_rocm_manifest.json 20737916627-linux/manifests/gfx94X-dcgpu/therock_torch_manifest.json 20737916627-linux/manifests/gfx94X-dcgpu/therock_jax_manifest.json

I think there is an issue when we have single name as therock_torch_manifest.json per run. We need six different json files (json per job) from a particular run: e.g. https://github.com/ROCm/TheRock/actions/runs/20808841882/job/59768375462. So the last completed job will overwrite the therock_torch_manifest.json file.

When #1236 is solved, each release build should use matching commits. Until then, we can upload one manifest per job with a descriptive name. We'll at least be able to compare manifests across workflow runs and jobs then to see how often the commits used differ.

I see, sounds good to me.

Tracking the future issue in here: #2827

Actually, I think we should sequence this work differently.

Solve Release workflows do not freeze the commits they select #1236 first. Generate manifest files at the start of the release pipelines for all releases, all repositories (ROCm, PyTorch, JAX, etc.).

Trigger release builds based on those computed manifests.

That should be much cleaner architecturally. The build scripts won't need to know anything about manifests, they'll continue to operate as they currently do, based on whatever code has been checked out ahead of time by the workflows. We can add a mode to the checkout scripts that reads from the manifest as needed.

ScottTodd · 2026-01-07T00:28:46Z

+        if: ${{ github.repository_owner == 'ROCm' }}
+        run: |
+          aws s3 cp ${{ env.PACKAGE_DIST_DIR }}/manifests/ \
+            s3://${{ env.S3_BUCKET_ARTIFACTS }}/external-builds/pytorch/${{ github.run_id }}/${{ inputs.amdgpu_family }}/ \


Let's think carefully about the top level folder structure.

Here's an example Linux release package build: https://github.com/ROCm/TheRock/actions/runs/20737916627#summary-59538918041

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/logs/gfx94X-dcgpu/index.html

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/logs/gfx94X-dcgpu/build_time_analysis.html

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/index-gfx94X-dcgpu.html

https://therock-nightly-artifacts.s3.amazonaws.com/20737916627-linux/manifests/gfx94X-dcgpu/therock_manifest.json

Could we match that and write to ${{ github.run_id }}-{{ platform }}/manifests/{{ inputs.amdgpu_family }}/therock_torch_manifest.json?

For example:

20737916627-linux/manifests/gfx94X-dcgpu/therock_rocm_manifest.json 20737916627-linux/manifests/gfx94X-dcgpu/therock_torch_manifest.json 20737916627-linux/manifests/gfx94X-dcgpu/therock_jax_manifest.json

erman-gurses · 2026-01-08T20:36:19Z

Test Linux: https://github.com/ROCm/TheRock/actions/runs/20830926850
Test Windows: https://github.com/ROCm/TheRock/actions/runs/20830878062

erman-gurses · 2026-01-09T04:54:51Z

Linux Test: https://github.com/ROCm/TheRock/actions/runs/20840415276/job/59873391163

Example manifest files from test:
https://therock-dev-artifacts.s3.us-east-2.amazonaws.com/20840415276-linux/manifests/gfx94X-dcgpu/therock-manifest_torch_py3.11_release-2.7.json

Windows Test: https://github.com/ROCm/TheRock/actions/runs/20840400033/job/59873344569

Example manifest files from test:
https://therock-dev-artifacts.s3.us-east-2.amazonaws.com/20840400033-windows/manifests/gfx1151/therock-manifest_torch_py3.11_release-2.9.json

…ile I/O

erman-gurses · 2026-02-12T16:34:27Z

Tests: https://github.com/ROCm/TheRock/actions/runs/21955293350

Example from the test:
https://therock-dev-artifacts.s3.us-east-2.amazonaws.com/21955293350-linux/manifests/gfx110X-all/therock-manifest_torch_py3.11_nightly.json

{
  "pytorch": {
    "commit": "a2508a3a3258955ba020584e5bff9cb64a6e4b32",
    "repo": "https://github.com/pytorch/pytorch.git"
  },
  "pytorch_audio": {
    "commit": "5fde87141e1d48a4fe2d0ab6371bb109bcc0ff46",
    "repo": "https://github.com/pytorch/audio.git"
  },
  "pytorch_vision": {
    "commit": "95c251f3f39a90877a41ff60471031cb4db0aa38",
    "repo": "https://github.com/pytorch/vision.git"
  },
  "triton": {
    "commit": "f5955032dea6ca418ed2b1ac28a86dbb8f9e2869",
    "repo": "https://github.com/ROCm/triton.git"
  },
  "therock": {
    "commit": "6c743f7f8c4f3207ea65982e40f5845211b41f46",
    "repo": "https://github.com/ROCm/TheRock.git",
    "branch": "users/erman-gurses/add-pytorch-manifest"
  }
}

ScottTodd

Looking pretty good. A few remaining comments about file organization and code style.

ScottTodd · 2026-02-12T20:48:24Z

+            --pytorch-dir "external-builds/pytorch/pytorch" \
+            --pytorch-audio-dir "external-builds/pytorch/pytorch_audio" \
+            --pytorch-vision-dir "external-builds/pytorch/pytorch_vision" \
+            --triton-dir "external-builds/pytorch/triton"


Let's add apex now too. Fine to be a follow-up PR.

See 39d840b

erman-gurses · 2026-02-12T23:43:49Z

Test: https://github.com/ROCm/TheRock/actions/runs/21970407618
Example:
https://therock-dev-artifacts.s3.us-east-2.amazonaws.com/21970407618-linux/manifests/gfx110X-all/therock-manifest_torch_py3.13_release-2.10.json

{
  "pytorch": {
    "commit": "e77a6493f222212b11f9fc91510e15465d106a8b",
    "repo": "https://github.com/ROCm/pytorch.git",
    "branch": "release/2.10"
  },
  "pytorch_audio": {
    "commit": "5047768f2447d963dffc250b64a5d6c01afc84fa",
    "repo": "https://github.com/pytorch/audio"
  },
  "pytorch_vision": {
    "commit": "82df5f599578b383987510836bb05ea97dcc9669",
    "repo": "https://github.com/pytorch/vision"
  },
  "triton": {
    "commit": "244a4867743cd408b3cbbca704e4c28a80533491",
    "repo": "https://github.com/ROCm/triton.git"
  },
  "therock": {
    "commit": "26d0ca4c2f744b10434c80bb7720f08f07d3223d",
    "repo": "https://github.com/ROCm/TheRock.git",
    "branch": "users/erman-gurses/add-pytorch-manifest"
  }
}

CC: @ScottTodd

…ctions/tests

ScottTodd · 2026-02-13T02:54:07Z

We could also include these changes in https://github.com/ROCm/TheRock/blob/main/.github/workflows/build_portable_linux_pytorch_wheels_ci.yml (new workflow for CI), but the longer term plans will solve for that naturally:

In the "setup" job, compute a single manifests for all projects and upload it (or multiple manifests: one for ROCm, one for PyTorch, one for JAX, etc.)

In all jobs follow that, use the manifests to decide what to checkout and build

## Motivation Add Apex to the Pytorch Manifest file Follow up PR for #2547 See the comment here: #2547 (comment) ## Test Plan Test on CI ## Test Result The test runs: https://github.com/ROCm/TheRock/actions/runs/22070943413 Example Manifest from the test: ``` { "pytorch": { "commit": "f466ea7ac710374ce7066d9ead92e57dc42dc76f", "repo": "https://github.com/ROCm/pytorch.git", "branch": "release/2.7" }, "pytorch_audio": { "commit": "95c61b4168fc5133be8dd8c1337d929d066ae6cf", "repo": "https://github.com/pytorch/audio" }, "pytorch_vision": { "commit": "59a3e1f9f78cfe44cb989877cc6f4ea77c8a75ca", "repo": "https://github.com/pytorch/vision" }, "triton": { "commit": "95abd4daa02d789ed257178ba35b4ef2d746ca29", "repo": "https://github.com/ROCm/triton.git" }, "apex": { "commit": "7a57becffa31a4394369152e978fa6f8a0f005c2", "repo": "https://github.com/ROCm/apex" }, "therock": { "commit": "02024427089134e951fe4063100b93c28e7e01a8", "repo": "https://github.com/ROCm/TheRock.git", "branch": "users/erman-gurses/add-apex-to-pytorch-manifest" } } ``` Link: https://therock-dev-artifacts.s3.us-east-2.amazonaws.com/22070943413-linux/manifests/gfx94X-dcgpu/therock-manifest_torch_py3.10_release-2.7.json ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

github-project-automation Bot added this to TheRock Triage Dec 15, 2025

github-project-automation Bot moved this to TODO in TheRock Triage Dec 15, 2025

ScottTodd reviewed Dec 15, 2025

View reviewed changes

Comment thread .github/workflows/build_portable_linux_pytorch_wheels.yml Outdated

marbre reviewed Dec 19, 2025

View reviewed changes

erman-gurses marked this pull request as ready for review January 5, 2026 06:08

erman-gurses requested review from ScottTodd, amd-aakash and marbre January 5, 2026 06:13

erman-gurses moved this from TODO to In Progress in TheRock Triage Jan 5, 2026

erman-gurses linked an issue Jan 5, 2026 that may be closed by this pull request

Add Manifest Generation for PyTorch External Builds #2481

Closed

marbre reviewed Jan 5, 2026

View reviewed changes

marbre self-requested a review January 5, 2026 22:11

ScottTodd reviewed Jan 7, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

erman-gurses commented Jan 8, 2026

View reviewed changes

Comment thread .github/workflows/build_portable_linux_pytorch_wheels.yml Outdated

erman-gurses requested a review from ScottTodd January 8, 2026 20:41

erman-gurses mentioned this pull request Jan 8, 2026

After #1236 is solved - each release build should use matching commits #2827

Open

ScottTodd requested changes Jan 12, 2026

View reviewed changes

erman-gurses requested a review from ScottTodd January 15, 2026 14:31

ScottTodd mentioned this pull request Jan 15, 2026

Release workflows do not freeze the commits they select #1236

Open

erman-gurses force-pushed the users/erman-gurses/add-pytorch-manifest branch from d49c64c to 3cb6cab Compare January 21, 2026 08:22

erman-gurses added 5 commits January 21, 2026 00:26

Generate a manifest file for Pytorch builds

ee237a7

Fix typo

8eb1108

Move PyTorch manifest uploads to therock artifacts bucket

a4492f3

Refactor script to match pep format

998f52e

Add fallbacks for GitHub related variables

4d6fefb

erman-gurses and others added 10 commits February 11, 2026 11:28

Remove platform flag from workflow

cd44bab

Add fix run command call

7e71e74

Pass run_id to retrieve_bucket_info

11a2616

Fix permission issue

5507082

Fix permission issue

35c2f15

Test

905eb11

Test

ac00aee

Refactor generate_pytorch_manifest to separate dict generation from f…

d350cb3

…ile I/O

Reorder therock keys

1d491f0

Merge branch 'main' into users/erman-gurses/add-pytorch-manifest

b054bd6

erman-gurses requested a review from ScottTodd February 12, 2026 16:34

Add correct path for unittests

6c743f7

ScottTodd reviewed Feb 12, 2026

View reviewed changes

erman-gurses and others added 5 commits February 12, 2026 13:38

Move files under build_tools/github_actions/ folder

f119e74

Remove extra platform checks

29c115c

Add branch info for the pytorch repos with best effort

76ca97b

Refactor generate_pytorch_manifest tests to mock git_head()

5aa2b7c

Merge branch 'main' into users/erman-gurses/add-pytorch-manifest

6a32a71

erman-gurses requested review from ScottTodd and marbre February 12, 2026 23:45

erman-gurses added 2 commits February 12, 2026 15:45

Relocate generate_pytorch_manifest_test.py under build_tools/github_a…

03cecce

…ctions/tests

Update the script path from the workflows

26d0ca4

ScottTodd approved these changes Feb 13, 2026

View reviewed changes

erman-gurses merged commit 28ae028 into main Feb 13, 2026
92 of 95 checks passed

github-project-automation Bot moved this from In Progress to Done in TheRock Triage Feb 13, 2026

erman-gurses deleted the users/erman-gurses/add-pytorch-manifest branch February 13, 2026 02:57

erman-gurses mentioned this pull request Feb 16, 2026

Add Apex to the Pytorch Manifest file #3431

Merged

1 task

Conversation

erman-gurses commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erman-gurses commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erman-gurses commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marbre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erman-gurses Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erman-gurses commented Jan 8, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

erman-gurses commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erman-gurses commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ScottTodd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erman-gurses commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

erman-gurses commented Dec 15, 2025 •

edited

Loading

erman-gurses commented Jan 5, 2026 •

edited

Loading

erman-gurses commented Jan 5, 2026 •

edited

Loading

erman-gurses Jan 8, 2026 •

edited

Loading

erman-gurses commented Jan 9, 2026 •

edited

Loading

erman-gurses commented Feb 12, 2026 •

edited

Loading

erman-gurses commented Feb 12, 2026 •

edited

Loading