feat: Added subgroups in pytorch jobs #935
Conversation
itsomri
force-pushed
the
omric/pytorch-subgrouping
branch
from
b3330df to
c25da43
Compare
itsomri
force-pushed
the
omric/pytorch-subgrouping
branch
from
5d9476f to
9ad48a5
Compare
Merging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
1 similar comment
Merging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
Merging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
itsomri
force-pushed
the
omric/pytorch-subgrouping
branch
from
8e0c415 to
7794422
Compare
Merging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
| ReplicaTypeLabel = pytorchv1.ReplicaTypeLabel | ||
|
|
||
| ReplicaTypeMaster = pytorchv1.PyTorchJobReplicaTypeMaster | ||
| ReplicaTypeWorker = pytorchv1.PyTorchJobReplicaTypeWorker |
There was a problem hiding this comment.
What do you think about making these private?
| masterReplicas = 0 | ||
| } | ||
|
|
||
| workerReplicas := totalMinAvailable - int32(masterReplicas) |
There was a problem hiding this comment.
What happens if totalMinAvailable < masterReplicas?
| subGroups = append(subGroups, &podgroup.SubGroupMetadata{ | ||
| Name: replicaType, | ||
| MinAvailable: minAvailable, | ||
| PodsReferences: podReferences, |
There was a problem hiding this comment.
Shouldn't we aggregate all this replica type pod references?
There was a problem hiding this comment.
PodReferences are aggregated: they're used by the podgroup handler to assign pods to subgroups - once a pod is assigned, there's no need to re-assign it. Doing so will require us to list the pods in the namespace and iterate over them. https://github.com/NVIDIA/KAI-Scheduler/blob/main/pkg/podgrouper/podgroup/handler.go#L143
3 participants
Description
Implemented subgroups in pytorch:
Related Issues
Fixes #
Checklist
Breaking Changes
Additional Notes