RFC for adding memory monitoring to debug OOM

[dezhiAmd]

Motivation

Self-hosted GitHub runners executing TheRock builds have been experiencing out-of-memory errors, causing build failures and CI instability. Without detailed memory usage tracking across different build phases, it is difficult to:

1. Identify the root cause of OOM failures
2. Determine which build phases consume the most memory
3. Optimize resource allocation and parallel job configurations
4. Proactively detect memory pressure before failures occur
5. Analyze historical trends in memory consumption

Technical Details

Test Plan

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[geomin12]

overall looks great! just a few housekeeping comments

[geomin12]

lgtm! let's get Shiraz to do a pass on this as well

[amd-shiraz]

@dezhiAmd are we generating any report post this operation ? if yes, can you pls share an example ? First pass through RFC i have some more questions, i will set up a meeting tomorrow to go over and understand. that will be better i believe. thanks!

[ScottTodd]

As mentioned multiple times during code reviews, these workflows are not an exhaustive list and we need a solution that scales to multiple build and release workflows, perhaps across multiple repositories. This current design requires deep changes to many files and substantial plumbing. Let's spend more time exploring what we can do at the machine/runner/cloud project level.

[dezhiAmd]

I believe this solution is scalable.
Please refer to the link where the component test is enabled for memory monitoring.
Even when leveraging an existing tool, some modifications to the current workflow are still required to capture the necessary metrics.
To support multiple repositories, the monitoring script can be added to each repository, followed by updates to the respective workflow files.
Key advantage: This approach provides a unified solution with full control, regardless of whether the runner is on-premises or hosted on cloud VMs.

CC @amd-shiraz @geomin12 @amd-justchen

[ScottTodd]

Workflows are moving towards this style: https://github.com/ROCm/TheRock/blob/main/.github/workflows/multi_arch_build_portable_linux.yml, with 7+ setup/build/report jobs (per platform). Any monitoring will need to handle multiple build stages and will need to integrate with those sorts of workflows with minimal plumbing.

Some ideas:

• Workflow-level environment variable loaded from inputs, read that env var during a script like configure_stage.py
• Instrument on the runners themselves
• Reusable workflow for "setup ccache", "runner health status", "enable monitoring", etc. (similar to what pytorch does for workflow init)

[amd-shiraz]

@dezhiAmd I have spent sometime thinking more on this one and as discussed offline last week here is my update and request.

1. real time tracking this metric is fine but perhaps integrating to the build is not.
2. Taking a step back and thinking through, if we are tracking overall runner health the way it should be then we don't need to introduce something as part of the build to track it as well.
3. this tool itself will not prevent the build from failing due to lets say OOM issues etc. but will help debug and triage those failures after the fact.
4. Mainly because of step 3, i would encourage us to use this tool as something we run as an external debug tool as and when needed Vs integrating as part of the build itself. eg: when the build failed due to OOM issue and we want to find out which step of the build caused it, we make use of this tool to be ran separately making use of the build logs + timestamps etc. to check.
5. nit: for RFCs like these that are more related to underlying infra, i would encourage us to update on TheRock-infra repo https://github.com/ROCm/TheRock-Infra/tree/main/docs/RFCs .
   Hope that sounds helpful and we can make some modifications accordingly. overall good stuff.
   cc @ScottTodd @geomin12

[marbre]

We had an issue with a history rewrite which confused the GH UI and the commits it is showing on your PR. Please make sure to update your branch e.g. in the GH UI via

Afterwards you need update your local branch via git pull.

Alternative git rebase solution

Alternatively you can manually update your branch locally by following the below instructions:

git checkout <yourbranch>
git fetch origin main
git rebase origin/main
git push --force-with-lease origin <yourbranch>

[ScottTodd]

Marking as 'reviewed' to drop this from my queue (need other reviewers to be on top of PRs like this too, I can't be the only one reviewing after a batch of comments)

[geomin12]

i am taking over this PR once i get some cycles

[geomin12]

Closing for #4050