Skip to content

Introduce Megatron-style parallel state management #7726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eternalNight wants to merge 24 commits into
base: master
Choose a base branch
from
+2,709 −0
Open

Introduce Megatron-style parallel state management #7726

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
efee3ef
Introduce Megatron-style parallel state management
Dec 15, 2025
e319923
parallel_state: Cleanup dependency on ProcessGroupNCCL.Options
eternalNight Jan 7, 2026
684f096
feat: add config-based parallel state initialization with validation
Dec 23, 2025
4b362e4
Add sequence parallel support to refactored parallel state
Jan 15, 2026
703c1fe
fix: remove Chinese comment from config example
Jan 19, 2026
04f56a9
fix: use torch.distributed.new_group directly in _create_group
Jan 19, 2026
845bd81
fix: correct SP parallel group creation logic in parallel_state
Jan 19, 2026
2ec5c69
refactor: simplify _create_group to use deepspeed.comm interface
Jan 21, 2026
33a3ca8
feat: migrate All-to-All groups to parallel_state architecture
Jan 21, 2026
138fbeb
fix: disable gloo process groups by default
Jan 23, 2026
2aab5b8
refactor: simplify SP group creation using RankGenerator
Jan 23, 2026
a0975c7
docs: fix config example and SP usage notes
Jan 26, 2026
58f7f51
refactor: remove unused is_torch_min_version function
Jan 26, 2026
ff67ec5
refactor: simplify config-based initialization to use top-level fields
Jan 26, 2026
11e5a2c
tests: Drop test_mpu.py from the PR
eternalNight Jan 26, 2026
f7bd2dc
fix: filter unsupported params in initialize_parallel_state_from_conf…
Feb 11, 2026
537a899
fix: improve initialize_parallel_state_from_config for nested config …
Mar 2, 2026
856ed67
docs: add parallel state management documentation
Mar 2, 2026
1084757
Fix ParallelState mpu compatibility and refactor unit tests
Mar 4, 2026
e74fe73
refactor: unify sequence-data parallel API naming
Mar 4, 2026
445b71e
Rename parallel_state_deepspeed to parallel_state_wrappers
Mar 4, 2026
da15a8a
parallel_state: Take parallelism sizes from existing parameters only
Mar 4, 2026
b1e16c4
tests: Test ZeRO + Ulysses SP training using ParallelState
Mar 4, 2026
5fbbb01
Merge branch 'master' into eternalNight/unify_process_group_management
eternalNight Mar 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading