Make GPU manager skip records when nothing scheduled on input stream #681
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bosilca
reviewed
Oct 16, 2024
Contributor
bosilca
left a comment
bosilca
left a comment
There was a problem hiding this comment.
By this logic all submission functions should now return a positive number, including the DTD GPU submission, the coroutines and the all the user-provided stage_in and out functions.
Contributor
|
@therault pretty please :) |
therault
force-pushed
the
gpu-skip-records-when-nothing-scheduled-on-stream
branch
from
71edab8 to
385a251
Compare
therault
force-pushed
the
gpu-skip-records-when-nothing-scheduled-on-stream
branch
from
c740700 to
22d8b8a
Compare
When a GPU kernel had all its data already on the GPU, we would still schedule a record event on the input stream. That would delay the scheduling of the kernel after all asyncMemcpy that had been scheduled before complete their execution, reducing parallelism between execution and I/O. With this change, the aim is to entirely skip recording the event on the input stream and directly schedule the ready-to-execute kernel on an exec stream of the GPU. Note that the behavior of data already in transfer is unchanged: no additional transfer is scheduled, but the task needs to wait on the input stream progress to be ready to execute.
… and don't forget to call complete_stage
…lready in transfer from CPU to GPU, fallback on scheduling another copy from the CPU if the transfer is yet incomplete.
…ng event while doing nothing to the GPU seems dubious to me)
therault
force-pushed
the
gpu-skip-records-when-nothing-scheduled-on-stream
branch
from
22d8b8a to
78a9749
Compare
bosilca
reviewed
Mar 28, 2025
devreal
approved these changes
Apr 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the GPU manager systematically records 3 polling events (or more for TTG) per task: one after kernel_push, one after kernels submission and one after kernel_pop. The event record after the kernel_push is needed if any input data is not yet on the GPU, even if that task did not schedule an actual cudaMemcpyAsync, because the cudaMemcpyAsync it depends on is already executing, and we use the serialization of the input stream to guarantee order.
However, in many cases (typically O(nt^3-nt^2) tasks for a GEMM that fits on GPU memory, or for a POTRF that fits in GPU memory), all data are already on the GPU. In that case, recording an event and waiting for it to exit the queue can be unnecessary overhead.
This PR skips event recording after the input stream if all input data is already on the GPU.
Pro:
Con:
@QingleiCao has tested the PR with variable tile size, and some benefit is observed for small tiles. No effect (detrimental or beneficial) is observed for large tiles on DPLASMA. @devreal wanted to test with TTG, where this is expected to bring more benefit (very small work in many tasks) and this PR should simplify him testing the combination of multiple branches.