Offload activation async support#156

Open

MayDomine wants to merge 144 commits intoOpenBMB:devfrom

MayDomine:tp_offload

Commits on Jul 24, 2023

using hooks to implement ZeRO and Checkpoint
zhangkaihuo
committed

Commits on Jul 25, 2023

async backward
zhangkaihuo
committed
async forward
zhangkaihuo
committed
merge upstream
zhangkaihuo
committed
fix
zhangkaihuo
committed

Commits on Jul 26, 2023

save cuda_rng_state
zhangkaihuo
committed

Commits on Jul 27, 2023

fix
zhangkaihuo
committed
fix
zhangkaihuo
committed
fix
zhangkaihuo
committed

Commits on Jul 31, 2023

remove __call__
zhangkaihuo
committed
refactor code structure
zhangkaihuo
committed
pipeline
zhangkaihuo
committed
for low version
zhangkaihuo
committed
for low torch version
zhangkaihuo
committed
for checkpoint
zhangkaihuo
committed
remove unused code
zhangkaihuo
committed
remove duplicate code
zhangkaihuo
committed

Commits on Aug 1, 2023

fix pipeline; checkpoint support low version
zhangkaihuo
committed
fix pipeline; checkpoint support low version
zhangkaihuo
committed
merge remote
zhangkaihuo
committed
fix indent
zhangkaihuo
committed

Commits on Aug 2, 2023

pipe support low version
zhangkaihuo
committed
custom linear for zero3
zhangkaihuo
committed

Commits on Aug 3, 2023

merge origin
zhangkaihuo
committed
resolve conflict
zhangkaihuo
committed
resolve conflict
zhangkaihuo
committed
use torch.utils.checkpoint.checkpoint
zhangkaihuo
committed

Commits on Aug 4, 2023

custom hook
zhangkaihuo
committed
optimize code structure
zhangkaihuo
committed
for hidden_state
zhangkaihuo
committed
for input.requires_grad is False
zhangkaihuo
committed

Commits on Aug 5, 2023

fix
zhangkaihuo
committed

Commits on Aug 6, 2023

pipeline support return hidden_state
zhangkaihuo
committed

Commits on Aug 7, 2023

fix args
zhangkaihuo
committed
fix test
zhangkaihuo
committed

Commits on Aug 8, 2023

CheckpointBlock -> BMTBlock
zhangkaihuo
committed
reset block name
zhangkaihuo
committed
pipeline support batch_related
zhangkaihuo
committed

Commits on Aug 9, 2023

remove use_checkpoint from init_distributed
zhangkaihuo
committed

Commits on Aug 10, 2023

test
zhangkaihuo
committed
test for transformer and attn
MayDomine
committed
for requires_grad
zhangkaihuo
committed
for requires_grad
zhangkaihuo
committed
fix for arg is not tensor
zhangkaihuo
committed
fix for arg is not a tensor
zhangkaihuo
committed
add test
zhangkaihuo
committed
Merge branch 'hook' of https://github.com/zkh2016/BMTrain into hook
zhangkaihuo
committed

Commits on Aug 11, 2023

merge enhance_ckp
zhangkaihuo
committed
enhance ckp
zhangkaihuo
committed
Merge branch 'hook' into test
zhangkaihuo
committed
test
zhangkaihuo
committed
test
zhangkaihuo
committed

Commits on Aug 12, 2023

test
zhangkaihuo
committed
refactor code
zhangkaihuo
committed
mv linear to bmt.nn.linear
zhangkaihuo
committed
for enhance_ckp
zhangkaihuo
committed

Commits on Aug 14, 2023

fix for all input not grad
zhangkaihuo
committed
fix pre_module
zhangkaihuo
committed
fix pre_module
zhangkaihuo
committed
fix for all input no grad
zhangkaihuo
committed
fix for all input no grad
zhangkaihuo
committed

Commits on Aug 15, 2023

Commits on Aug 16, 2023

Commits on Aug 17, 2023

offloading bug fix
MayDomine
committed
fix reentrant
zhangkaihuo
committed
Merge branch 'hook' of https://github.com/zkh2016/BMTrain into hook
zhangkaihuo
committed

Commits on Aug 18, 2023

Merge branch 'dev' into hook
zhangkaihuo
authored

Commits on Aug 20, 2023

refactor CheckpointBlock
zhangkaihuo
committed
refactor pipe
zhangkaihuo
committed
Merge branch 'hook' of https://github.com/zkh2016/BMTrain into hook
zhangkaihuo
committed
fix all input no grad
zhangkaihuo
committed
fix hiddenstate
zhangkaihuo
committed

Commits on Aug 21, 2023

fix test
zhangkaihuo
committed
fix
zhangkaihuo
committed
remove unused import
zhangkaihuo
committed
fix pre_module
zhangkaihuo
committed
recovery some code
zhangkaihuo
committed
add test_no_grad.py
zhangkaihuo
committed
test unroll block list
zhangkaihuo
committed
fix test_fp32
zhangkaihuo
committed
cross_entropy support fp32
zhangkaihuo
committed
offload context
MayDomine
committed
Merge branch 'hook' into test
MayDomine
committed
cpm live for offloading test
MayDomine
committed

Commits on Aug 22, 2023

Better hack for offload
MayDomine
committed
fix OFFLOAD _mode bug
MayDomine
committed
fix is_first_layer
zhangkaihuo
committed

Commits on Aug 23, 2023

Fix async bug
MayDomine
committed
tensor parallel
zhangkaihuo
committed
Merge branch 'fix_first_layer' into tensor_parallel
zhangkaihuo
committed
rm unused code
zhangkaihuo
committed

Commits on Aug 24, 2023

refactor nccl group; remove partition_modules in pipe_layer.py
zhangkaihuo
committed
fix by review comment
zhangkaihuo
committed
fix topology
zhangkaihuo
committed
offload event wait
MayDomine
committed
Merge branch 'dev' of https://github.com/OpenBMB/BMTrain into offload
MayDomine
committed
fix topology
zhangkaihuo
committed
fix
zhangkaihuo
committed
use ParallelEmbedding
zhangkaihuo
committed
Offload Correct Version
MayDomine
committed
overlap parallel linear backward
zhangkaihuo
committed
add tp_comm_stream
zhangkaihuo
committed
fix tp
Achazwl
committed
Merge branch 'tensor_parallel' into tp
Achazwl
authored
Merge pull request #1 from Achazwl/tp
zhangkaihuo
authored

Commits on Aug 25, 2023

new hook storage
MayDomine
committed
Offload storage function fix
MayDomine
committed
storage dont release fix
MayDomine
committed
fix load_state_dict
zhangkaihuo
committed
test parallel linear
zhangkaihuo
committed
mv zero_level to CheckpointBlock
zhangkaihuo
committed
use dataptr as storage id
MayDomine
committed
merge dev
zhangkaihuo
committed
Merge with dev
MayDomine
committed
fix prev confilct
MayDomine
committed
fix overlap
zhangkaihuo
committed
gather once in atten
zhangkaihuo
committed
fix sub grad_input in parallel linear
zhangkaihuo
committed

Commits on Aug 26, 2023

Commits on Aug 29, 2023