Skip to content

feat: add GPU time-slicing support via virtual devices#189

Open
andrei-indreies wants to merge 1 commit into
ROCm:masterfrom
andrei-indreies:feat/add-time-slicing
Open

feat: add GPU time-slicing support via virtual devices#189
andrei-indreies wants to merge 1 commit into
ROCm:masterfrom
andrei-indreies:feat/add-time-slicing

Conversation

@andrei-indreies
Copy link
Copy Markdown

Motivation

Add GPU time-slicing support to the ROCm Kubernetes Device Plugin.

I have an r5 4650g node inside my k3s on which I am want to run both Jellyfin and tdarr, but with the current state of the plugin is not doable because the plugin strictly registers each physical AMD GPU exactly once, providing no built-in way for multiple low-resource pods to share a single GPU. This PR introduces Kubernetes-level overcommit (time-slicing).

Technical Details

Added --replicas integer flag to cmd/k8s-device-plugin/main.go (defaults to 1, preserving existing behavior).

  • Created internal/pkg/plugin/timeslice.go with helpers to build and resolve virtual device IDs (appending and stripping -slice-<n> suffixes).
  • ListAndWatch: Now multiplies discovered physical GPUs by replicas, advertising virtual IDs like 0000:03:00.0-slice-0 to Kubernetes.
  • Allocate: Resolves virtual device IDs back to their parent physical IDs to correctly assign hardware topology and /dev/dri/* device paths.
  • Health Checks: Updated PopulatePerGPUDHealth in exporter/health.go to optionally accept a physical ID resolver so virtual slices inherit their parent GPU's health status.
  • Added comprehensive unit tests in timeslice_test.go covering edge cases.
  • Updated README.md to document the time-slicing feature, how to configure it, and its hardware isolation caveats.

Test Plan

  1. Unit tests: Run go test for internal/pkg/plugin/ to verify virtual device creation and physical ID resolution logic.
  2. Cluster deployment:
    • Deploy the plugin DaemonSet on a node with 1 physical AMD GPU using --replicas=4.
    • Verify kubectl get node reports 4 for amd.com/gpu capacity.
    • Deploy 3 independent test pods, each requesting amd.com/gpu: 1.
    • Verify all 3 pods successfully schedule onto the same node simultaneously.
    • Exec into the pods to verify /dev/kfd and /dev/dri/renderD* are correctly mounted.

Test Result

  • go test passes successfully for all new time-slicing test cases.
  • Docker image builds successfully.
  • Node capacity successfully reflects the multiplied virtual GPU count.
  • Multiple pods successfully scheduled onto a single node, overcoming previous 1:1 hardware limitations. Device bind-mounts confirmed working inside the containers.
  • Production case: I transcoded a 10 bit movie in the same time tdarr converted a h264 movie in a h265

Submission Checklist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant