Introduce Megatron-style parallel state management#7726

Open

eternalNight wants to merge 24 commits intodeepspeedai:masterfrom

openanolis:eternalNight/unify_process_group_management

Commits on Mar 4, 2026

Introduce Megatron-style parallel state management

言枢
authored and
Junjie Mao
committed
parallel_state: Cleanup dependency on ProcessGroupNCCL.Options

eternalNight
authored and
Junjie Mao
committed
feat: add config-based parallel state initialization with validation

言枢
authored and
Junjie Mao
committed
Add sequence parallel support to refactored parallel state

yunqing
authored and
Junjie Mao
committed
fix: remove Chinese comment from config example

言枢
authored and
Junjie Mao
committed
fix: use torch.distributed.new_group directly in _create_group

言枢
authored and
Junjie Mao
committed
fix: correct SP parallel group creation logic in parallel_state

yunqing
authored and
Junjie Mao
committed
refactor: simplify _create_group to use deepspeed.comm interface

言枢
authored and
Junjie Mao
committed
feat: migrate All-to-All groups to parallel_state architecture

言枢
authored and
Junjie Mao
committed
fix: disable gloo process groups by default

言枢
authored and
Junjie Mao
committed
refactor: simplify SP group creation using RankGenerator

yunqing
authored and
Junjie Mao
committed
docs: fix config example and SP usage notes

言枢
authored and
Junjie Mao
committed
refactor: remove unused is_torch_min_version function

言枢
authored and
Junjie Mao
committed
refactor: simplify config-based initialization to use top-level fields

言枢
authored and
Junjie Mao
committed
tests: Drop test_mpu.py from the PR

eternalNight
authored and
Junjie Mao
committed
fix: filter unsupported params in initialize_parallel_state_from_config and add integration tests

yunqing
authored and
Junjie Mao
committed
fix: improve initialize_parallel_state_from_config for nested config and usability

言枢
authored and
Junjie Mao
committed
docs: add parallel state management documentation

言枢
authored and
Junjie Mao
committed
Fix ParallelState mpu compatibility and refactor unit tests

言枢
authored and
Junjie Mao
committed
refactor: unify sequence-data parallel API naming

yunqing
authored and
Junjie Mao
committed
Rename parallel_state_deepspeed to parallel_state_wrappers
Junjie Mao
committed
parallel_state: Take parallelism sizes from existing parameters only
Junjie Mao
committed
tests: Test ZeRO + Ulysses SP training using ParallelState
Junjie Mao
committed
Merge branch 'master' into eternalNight/unify_process_group_management
eternalNight
authored

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Megatron-style parallel state management#7726

Introduce Megatron-style parallel state management#7726
eternalNight wants to merge 24 commits intodeepspeedai:masterfrom
openanolis:eternalNight/unify_process_group_management

Commits on Mar 4, 2026

Introduce Megatron-style parallel state management

parallel_state: Cleanup dependency on ProcessGroupNCCL.Options

feat: add config-based parallel state initialization with validation

Add sequence parallel support to refactored parallel state

fix: remove Chinese comment from config example

fix: use torch.distributed.new_group directly in _create_group

fix: correct SP parallel group creation logic in parallel_state

refactor: simplify _create_group to use deepspeed.comm interface

feat: migrate All-to-All groups to parallel_state architecture

fix: disable gloo process groups by default

refactor: simplify SP group creation using RankGenerator

docs: fix config example and SP usage notes

refactor: remove unused is_torch_min_version function

refactor: simplify config-based initialization to use top-level fields

tests: Drop test_mpu.py from the PR

fix: filter unsupported params in initialize_parallel_state_from_config and add integration tests

fix: improve initialize_parallel_state_from_config for nested config and usability

docs: add parallel state management documentation

Fix ParallelState mpu compatibility and refactor unit tests

refactor: unify sequence-data parallel API naming

Rename parallel_state_deepspeed to parallel_state_wrappers

parallel_state: Take parallelism sizes from existing parameters only

tests: Test ZeRO + Ulysses SP training using ParallelState

Merge branch 'master' into eternalNight/unify_process_group_management