Variable batch size and LR scheduler (moved to #7104)#5237

Closed

bm-synth wants to merge 588 commits intodeepspeedai:masterfrom

bm-synth:variable_batch_size_and_lr

This pull request is big! We're only showing the most recent 250 commits

Commits on Feb 8, 2025

Update linear.py compatible with torch 2.4.0 (#5811)

authored and
bm-synth
committed
GDS Swapping Fix (#6386)

authored and
bm-synth
committed
Long sequence parallelism (Ulysses) integration with HuggingFace (#5774)

authored and
bm-synth
committed
reduce cpu host overhead when using moe (#5578)

authored and
bm-synth
committed
fix fp16 Qwen2 series model to DeepSpeed-FastGen (#6028)

authored and
bm-synth
committed
Add Japanese translation of Windows support blog (#6394)

tohtana
authored and
bm-synth
committed
Correct op_builder path to xpu files for trigger XPU tests (#6398)

loadams
authored and
bm-synth
committed
add pip install cutlass version check (#6393)

GuanhuaWang
authored and
bm-synth
committed
[XPU] API align with new intel pytorch extension release (#6395)

authored and
bm-synth
committed
Update to prepare for next release

loadams
authored and
bm-synth
committed
Pydantic v2 migration (#5167)

authored and
bm-synth
committed
Fix torch check (#6402)

loadams
authored and
bm-synth
committed
Update version.txt after 0.15.0 release (#6403)

loadams
authored and
bm-synth
committed
Fix Type Mismatch (#6410)

authored and
bm-synth
committed
Fix redundant seq data parallel grp argument in Z3/MiCS (#5352)

authored and
bm-synth
committed
add Huawei Ascend NPU setup guide (#6445)

authored and
bm-synth
committed
Revert "Fix torch check (#6402)"

loadams
authored and
bm-synth
committed
Revert "Revert "Fix torch check (#6402)""

loadams
authored and
bm-synth
committed
Add documentation for launcher without SSH (#6455)

authored and
bm-synth
committed
Dtype support check for accelerator in UTs (#6360)

authored and
bm-synth
committed
Store/Load CIFAR from local/offline (#6390)

authored and
bm-synth
committed
Add the accelerator setup guide link in Getting Started page (#6452)

rogerxfeng8
authored and
bm-synth
committed
Allow triton==3.0.x for fp_quantizer (#6447)

siddartha-RE
authored and
bm-synth
committed
Change GDS to 1 AIO thread (#6459)

authored and
bm-synth
committed
[CCL] fix condition issue in ccl.py (#6443)

authored and
bm-synth
committed
Avoid gds build errors on ROCm (#6456)

authored and
bm-synth
committed
TestLowCpuMemUsage UT get device by device_name (#6397)

authored and
bm-synth
committed
Add workflow to build DS without torch to better test before releases (#6450)

loadams
authored and
bm-synth
committed
bf16_optimizer: fixes to different grad acc dtype (#6485)

nelyahu
authored and
bm-synth
committed
DeepNVMe tutorial (#6449)

authored and
bm-synth
committed
print warning if actual triton cache dir is on NFS, not just for default (#6487)

jrandall
authored and
bm-synth
committed
Fix patch for parameter partitioning in zero.Init() (#6388)

authored and
bm-synth
committed
DS_BUILD_OPS should build only compatible ops (#6489)

tjruwase
authored and
bm-synth
committed
Add default value to "checkpoint_folder" in "load_state_dict" of bf16_optimizer (#6446)

authored and
bm-synth
committed
Safe usage of popen (#6490)

tjruwase
authored and
bm-synth
committed
Handle an edge case where `CUDA_HOME` is not defined on ROCm systems (#6488)

amorehead
authored and
bm-synth
committed
Update version.txt after 0.15.1 release (#6493)

loadams
authored and
bm-synth
committed
HPU: add required ENV vars to acccelerator init (#6495)

authored and
bm-synth
committed
Op_builder->is_compatible quite warning (#6093)

authored and
bm-synth
committed
fix pipeline eval_batch micro_batches argument for schedule (#6484)

authored and
bm-synth
committed
Fix the broken url link (#6500)

rogerxfeng8
authored and
bm-synth
committed
fix environment variable export bug for MultiNodeRunner (#5878)

authored and
bm-synth
committed
Revert "BF16 optimizer: Clear lp grads after updating hp grads in hook" (#6508)

nelyahu
authored and
bm-synth
committed
wrap include cuda_bf16.h with ifdef BF16_AVAILABLE (#6520)

oelayan7
authored and
bm-synth
committed
Avoid security issues of subprocess shell (#6498)

authored and
bm-synth
committed
Add conditional on torch version for scaled_dot_product_attention (#6517)

loadams
authored and
bm-synth
committed
Added Intel Gaudi to Accelerator Setup Guide (#6543)

authored and
bm-synth
committed
Skip failing newly added tests in accelerate (#6574)

loadams
authored and
bm-synth
committed
Use msgpack for p2p comm (#6547)

authored and
bm-synth
committed
DeepNVMe perf tuning (#6560)

authored and
bm-synth
committed
[Accelerator] Cambricon MLU support (#6472)

authored and
bm-synth
committed
Fix gradient accumulation for Z2+offload (#6550)

authored and
bm-synth
committed
fix errors when setting zero3 leaf modules with torch.compile (#6564)

authored and
bm-synth
committed
[XPU] Support DeepNVMe new code structure (#6532)

authored and
bm-synth
committed
Add APIs to offload states of model, optimizer, and engine (#6011)

authored and
bm-synth
committed
add bfloat16 to inference support dtypes (#6528)

authored and
bm-synth
committed
[COMPILE] workflow for deepspeed + torch.compile (#6570)

authored and
bm-synth
committed
Fixes on the accelerate side mean we do not need to skip this test (#6583)

loadams
authored and
bm-synth
committed
Fix torch include in `op_builder/mlu/fused_adam.py` and update no-torch workflow triggers (#6584)

loadams
authored and
bm-synth
committed
[ROCm] Fix subprocess error (#6587)

jagadish-amd
authored and
bm-synth
committed
Cleanup CODEOWNERS file to be valid (#6603)

loadams
authored and
bm-synth
committed
Add SSF Best practices badge (#6604)

loadams
authored and
bm-synth
committed
Move V100 workflows from cuda 11.1/11.7 to 12.1 (#6607)

loadams
authored and
bm-synth
committed
Fix SD workflow (#6609)

loadams
authored and
bm-synth
committed
Pin accelerate to fix CI failures/issues (#6610)

loadams
authored and
bm-synth
committed
Add llama3.2 vision autotp (#6577)

authored and
bm-synth
committed
Improve DS logging control (#6602)

tjruwase
authored and
bm-synth
committed
Fix device selection using CUDA_VISIBLE_DEVICES (#6530)

authored and
bm-synth
committed
Handle when `backend` is also in compile_kwargs (#6502)

authored and
bm-synth
committed
Rearrange inference OPS and stop using builder.load (#5490)

authored and
bm-synth
committed
Unpin accelerate tests, update lightning with node16 removal. (#6611)

loadams
authored and
bm-synth
committed
Enabled Qwen2-MoE Tensor Parallelism (TP) inference (#6551)

authored and
bm-synth
committed
Update version.txt after 0.15.2 release (#6615)

authored and
bm-synth
committed
Clean up prefetched parameters (#6557)

authored and
bm-synth
committed
AIO CPU Locked Tensor (#6592)

authored and
bm-synth
committed
reduce setting global variables to reduce torch compile graph breaks (#6541)

authored and
bm-synth
committed
Add API to get devices of offload states (#6586)

authored and
bm-synth
committed
apply fp16 autocast only to floating point values

Masahiro Tanaka
authored and
bm-synth
committed
Ignore reuse_dist_env (#6623)

tohtana
authored and
bm-synth
committed
[compile] Show breakdown of graph break (#6601)

delock
authored and
bm-synth
committed
Add API for updating ZeRO gradients (#6590)

tjruwase
authored and
bm-synth
committed
Accept btl_tcp_if_include option through launcher_args (#6613)

authored and
bm-synth
committed
Add first Step in LR Schedulers (#6597)

authored and
bm-synth
committed
Support safetensors export (#6579)

authored and
bm-synth
committed
add option to disable logger while compiling to avoid graph breaks (#6496)

authored and
bm-synth
committed
Lock cache file of HF model list (#6628)

tohtana
authored and
bm-synth
committed
Add README Pipeline Status for Huawei Ascend NPU (#6588)

authored and
bm-synth
committed
Update torch version in workflows (#6631)

tohtana
authored and
bm-synth
committed
Use file store for tests (#6632)

tohtana
authored and
bm-synth
committed
Fix Memory Leak In AIO (#6630)

authored and
bm-synth
committed
[XPU] upgrade xpu max1100 CI workflow to pytorch2.3 (#6646)

Liangliang-Ma
authored and
bm-synth
committed
[XPU] host timer check version from Torch 2.5 to Torch 2.6 (#6633)

authored and
bm-synth
committed
[XPU] [DeepNVMe] use same cpu_op_desc_t with cuda (#6645)

Liangliang-Ma
authored and
bm-synth
committed
Update version.txt after 0.15.3 release (#6652)

authored and
bm-synth
committed
Fix expert grad scaling problem with ZeRO optimizer (#6546)

authored and
bm-synth
committed
Add attribute check for language_model when replace last linear module (#6650)

authored and
bm-synth
committed
fix init_device_mesh for torch 2.4 (#6614)

authored and
bm-synth
committed
Fix dynamo issue (#6527)

authored and
bm-synth
committed
sequence parallel for uneven heads (#6392)

authored and
bm-synth
committed
Add fallback for is_compiling (#6663)

authored and
bm-synth
committed
Update profiler registration check (#6668)

loadams
authored and
bm-synth
committed
Add support for H100/sm_90 arch compilation (#6669)

loadams
authored and
bm-synth
committed
Update Gaudi2 docker image (#6677)

loadams
authored and
bm-synth
committed
Update gaudi2 docker version to latest release (1.18) (#6648)

authored and
bm-synth
committed
Update base docker image for A6000 GPU tests (#6681)

loadams
authored and
bm-synth
committed
Remove packages that no longer need to be updated in the latest container (#6682)

loadams
authored and
bm-synth
committed
Fix training of pipeline based peft's lora model (#5477)

authored and
bm-synth
committed
Update checkout action to latest version (#5021)

loadams
authored and
bm-synth
committed
Add attribute check to support git-base autotp (#6688)

authored and
bm-synth
committed
fix memcpy issue on backward for zero-infinity (#6670)

authored and
bm-synth
committed
Free memory in universal checkpointing tests (#6693)

authored and
bm-synth
committed
Explictly set device when reusing dist env (#6696)

tohtana
authored and
bm-synth
committed
Update URL in README Pipeline Status for Huawei Ascend NPU (#6706)

xuedinge233
authored and
bm-synth
committed
Pin transformers to 4.45.2 in nv-ds-chat workflow (#6710)

loadams
authored and
bm-synth
committed
[Bug Fix] Support threads_per_head < 64 for wavefront size of 64 (#6622)

authored and
bm-synth
committed
Use one param coordinator for both train/inference scenarios (#6662)

authored and
bm-synth
committed
Update yapf version (#6721)

loadams
authored and
bm-synth
committed
Update flake8 version (#6722)

loadams
authored and
bm-synth
committed
Switch what versions of python are supported (#5676)

authored and
bm-synth
committed
Update version.txt after 0.15.4 release (#6731)

loadams
authored and
bm-synth
committed
Update GH hosted workflows to 24.04 (#6717)

loadams
authored and
bm-synth
committed
Add COMMITTER file (#6741)

tjruwase
authored and
bm-synth
committed
Update AMD apex version (#6739)

loadams
authored and
bm-synth
committed
Fix Type Name Inconsistency & Typo in cpu_adam (#6732)

authored and
bm-synth
committed
Add Domino code (#6733)

authored and
bm-synth
committed
Add data type check for bf16 (#6742)

hwchen2017
authored and
bm-synth
committed
add zero3 ```module_granularity_threshold ``` to zero optimization. (#6649)

authored and
bm-synth
committed
AIO File Offsets (#6641)

authored and
bm-synth
committed
Update path for BingBertSquad from DeepSpeedExamples (#6746)

loadams
authored and
bm-synth
committed
Sanitize inputs to eval() (#6745)

loadams
authored and
bm-synth
committed
Adding the governance doc (#6748)

authored and
bm-synth
committed
Add no_sync context manager (#6675)

authored and
bm-synth
committed
Gaudi2 Nightly job for daily check (#6753)

authored and
bm-synth
committed
Disable failing python tests (#6758)

loadams
authored and
bm-synth
committed
A faster and more memory-efficient implementation of `zero_to_fp32` (#6658)

authored and
bm-synth
committed
Pin transformers version to work around latest torch requirements (#6759)

loadams
authored and
bm-synth
committed
make xpu ops compatible with oneapi 2025.0 (#6760)

authored and
bm-synth
committed
Add explicit parameters for torch.load (#6751)

loadams
authored and
bm-synth
committed
Fix setup.py bash cmd generation to correctly extract git info (#6762)

authored and
bm-synth
committed
Use `json_schema_extra` instead of extra keyword in `Field` (#6764)

authored and
bm-synth
committed
Enable torch compile on _allgather_params (#6769)

deepcharm
authored and
bm-synth
committed
Removes unnecessary cloning (#6761)

authored and
bm-synth
committed
Fix potential memory issues when use deepspeed Z3 (#6726)

authored and
bm-synth
committed
Unpin with latest transformers fixes (#6763)

loadams
authored and
bm-synth
committed
docs: fix HF links (#6780)

imba-tjd
authored and
bm-synth
committed
Fix Doc Error: ZeRO Stage 2 gradient partitioning (#6775)

yewentao256
authored and
bm-synth
committed
Cleanup code docs warnings (#6783)

loadams
authored and
bm-synth
committed
Domino Blog (#6776)

authored and
bm-synth
committed
Update version.txt before release (#6784)

loadams
authored and
bm-synth
committed
Revert release workflow (#6785)

loadams
authored and
bm-synth
committed
Update version.txt after 0.16.0 release (#6786)

loadams
authored and
bm-synth
committed
Domino news update on readme.md (#6815)

GuanhuaWang
authored and
bm-synth
committed
Fix zero checkpoint (#6792)

authored and
bm-synth
committed
Update python version but now we need to include setuptools on our own (#6787)

loadams
authored and
bm-synth
committed
Adding the new feature of FPDT (#6462)

authored and
bm-synth
committed
Pin transformers to avoid errors with latest version (#6820)

loadams
authored and
bm-synth
committed
Ulyssess offload blog (#6814)

authored and
bm-synth
committed
add FPDT tutorial (#6813)

authored and
bm-synth
committed
Update README.md (#6824)

samadejacobs
authored and
bm-synth
committed
Update README.md (#6825)

authored and
bm-synth
committed
Pin transformers version in cpu-torch-latest due to multiprocessing error. (#6823)

loadams
authored and
bm-synth
committed
Update pre-commit version (#6821)

loadams
authored and
bm-synth
committed
Update version.txt after 0.16.1 release (#6826)

loadams
authored and
bm-synth
committed
Pin HPU tests (#6831)

loadams
authored and
bm-synth
committed
Flops profiler support einops.einsum (#6755)

authored and
bm-synth
committed
Pin pytest-subtests version for accelerate tests (#6842)

loadams
authored and
bm-synth
committed
Inference UTs check for trition support from accelerator (#6782)

authored and
bm-synth
committed
Unpin pytest-subtests now that 0.14.1 is released (#6844)

loadams
authored and
bm-synth
committed
Merge LoCo with Zero++ (#6730)

authored and
bm-synth
committed
Fix type error in `ZeROOrderedDict` (#6794)

authored and
bm-synth
committed
Fix uneven head sequence parallelism bug (#6774) (#6797)

authored and
bm-synth
committed
Fix nv-torch-nightly test by pinning transformers (#6849)

loadams
authored and
bm-synth
committed
Remove broken links to non-active site (#6854)

authored and
bm-synth
committed
Avoid poisoning process with CUDA calls as soon as importing (#6810)

authored and
bm-synth
committed
Fix xpu tests workflow failure by changing pip index url (#6864)

Liangliang-Ma
authored and
bm-synth
committed
Domino updates (#6861)

authored and
bm-synth
committed
add domino navigation (#6866)

GuanhuaWang
authored and
bm-synth
committed
Update TSC (#6867)

tjruwase
authored and
bm-synth
committed
Remove warnings from autodoc and sphinx (#6788)

authored and
bm-synth
committed
Update real_accelerator.py (#6845)

authored and
bm-synth
committed
Fix assertion for offloading states (#6855)

authored and
bm-synth
committed
Remove pin from transformers version and fix Processing/Threading issues in tests (#6822)

loadams
authored and
bm-synth
committed
Add MLP/lm_head tp grain size setting. (#6828)

authored and
bm-synth
committed
Fix --enable_each_rank_log when used with PDSH multi-node runner (#6863)

authored and
bm-synth
committed
Update transformers ops unit tests to use `requried_torch_version` (#6884)

loadams
authored and
bm-synth
committed
Don't error out when cpu accelerator doesn't have torch (as default for whl building) (#6886)

loadams
authored and
bm-synth
committed
Add arctic model support by adding w2 to all_reduce (#6856)

authored and
bm-synth
committed
Update code owners (#6890)

authored and
bm-synth
committed
Update version.txt after 0.16.2 release (#6893)

loadams
authored and
bm-synth
committed
Allow to compile collective for PT>2.3 (#6899)

NirSonnenschein
authored and
bm-synth
committed
Zero2: avoid graph breaks in torch.compile by using param_idx (#6803)

authored and
bm-synth
committed
hpu_accelerator: use torch.use_deterministic_algorithms (#6897)

nelyahu
authored and
bm-synth
committed
Fix error caused by all_reduce call in domino (#6880)

authored and
bm-synth
committed
Update Gaudi2 jobs to latest 1.19 build (#6905)

authored and
bm-synth
committed
Change compile for pipeline module torch.compile (#6478)

authored and
bm-synth
committed
Stage3: Use new torch grad accumulation hooks API (#6773)

authored and
bm-synth
committed
Cleanup ops/transformer/inference tests (#6830)

loadams
authored and
bm-synth
committed
Fix `checkpointable_layers` Logic (#6881)

authored and
bm-synth
committed
[BUG FIX]:fix get torch.version.cuda error when cuda is None in rocm (#6909)

authored and
bm-synth
committed
Add fp8_gemm fallback for non-triton systems (#6916)

authored and
bm-synth
committed
Reduce the device bubble introduced by heavy loop synchronization in coalesced fetch/release(z3_leaf_module) (#6694)

authored and
bm-synth
committed
Cleanup ops/transformer/inference tests (#6925)

loadams
authored and
bm-synth
committed
Check transformers version in BLOOM for inference v1 (#6766)

authored and
bm-synth
committed
inference: remove unused _validate_args function (#5505)

authored and
bm-synth
committed
Use `torch.log1p` (#6930)

kit1980
authored and
bm-synth
committed
Update python version classifiers (#6933)

loadams
authored and
bm-synth
committed
Fix building on Windows with presence of Triton (#6749)

authored and
bm-synth
committed
Fix windows blog examples (#6934)

loadams
authored and
bm-synth
committed
Add deepseek autotp (#6937)

Yejing-Lai
authored and
bm-synth
committed
Add position_ids arg to OPTEmbedding forward function (#6939)

authored and
bm-synth
committed
Add information on security expectations with this software (#6941)

loadams
authored and
bm-synth
committed
Support pure meta model lm_head tp (#6812)

authored and
bm-synth
committed
Remove op compilation flags due to perf issue (#6944)

NirSonnenschein
authored and
bm-synth
committed
Pin nv-a6000 workflow (#6938)

loadams
authored and
bm-synth
committed
[inf] Add config var to enable keeping module on host (#6846)

authored and
bm-synth
committed
`warn` to `warning` (#6952)

qgallouedec
authored and
bm-synth
committed
Add extra_repr to Linear classes for debugging purpose (#6954)

Xia-Weiwen
authored and
bm-synth
committed
Update import for torchvision.transformers (#6958)

loadams
authored and
bm-synth
committed
Remove Duplicate Declaration of pandas in `Dockerfile` (#6959)

Zerohertz
authored and
bm-synth
committed
Add the missing view operations from sequence parallel(async). (#6750)

authored and
bm-synth
committed
Update `torch.norm` to `torch.linalg.norm` and `torch.linalg.vector_norm` (#6931)

loadams
authored and
bm-synth
committed
Using explicit GPU upcast for ZeRO-Offload (#6962)

xylian86
authored and
bm-synth
committed
Update version.txt after 0.16.3 release (#6965)

loadams
authored and
bm-synth
committed
Precisely track nvme optimizer offload (#6963)

tjruwase
authored and
bm-synth
committed
Update build_win.bat script to exclue GDS op as it lacks Windows support. (#6971)

loadams
authored and
bm-synth
committed
Add CUDA 12.8 support and comment on CUDA 12.7 (#6975)

loadams
authored and
bm-synth
committed
Update torch versions to support 2.6 (#6977)

loadams
authored and
bm-synth
committed
generalize deepspeed linear and implement it for non cuda systems (#6932)

authored and
bm-synth
committed
Update recommended Windows whl building versions (#6983)

loadams
authored and
bm-synth
committed
Title: Fix setup_env_ranks to Properly Set Environment Variables Instead of Raising Error (#6979)

authored and
bm-synth
committed
Specify torchvision in nv-ds-chat workflow (prevents errors with torch 2.6) (#6982)

loadams
authored and
bm-synth
committed
Remove assumption that padding only occurs on last rank (#6974)

authored and
bm-synth
committed
Use ds-specific module id to avoid conflicts (#6847)

authored and
bm-synth
committed
Update A6000 workflows to use newer docker container - 24.09 vs 24.03 (#6967)

loadams
authored and
bm-synth
committed
Allow NVIDIA Blackwell (#6991)

fabiendupont
authored and
bm-synth
committed
Update GH org references (#6998)

authored and
bm-synth
committed
Update CNAME

loadams
authored and
bm-synth
committed
Update CNAME

loadams
authored and
bm-synth
committed
[XPU] max1100 workflow update for docker and softwares (#7003)

Liangliang-Ma
authored and
bm-synth
committed
autotp training(fix dco) (#7004)

inkcherry
authored and
bm-synth
committed
import triton files when triton is supported and installed (#6989)

oelayan7
authored and
bm-synth
committed
Update A6000 tests transformers version (#7016)

authored and
bm-synth
committed
correct comment in DYNAMIC_BATCHING_SAMPLES_ORDER and DYNAMIC_BATCHING_SAMPLES_ORDER_DEFAULT
bm-synth
committed
removed unnecessary comment
bm-synth
committed
better comments
bm-synth
committed
fixed issue when self.steps_per_print() is None
bm-synth
committed
cleaner code
bm-synth
committed
cleaner config
bm-synth
committed
removed ruff errors on lambda expression
bm-synth
committed
cleanup after example
bm-synth
committed