Skip to content

Commit 01f520e

Browse files
authored
Fix README for LLaMA-2 fine-tuning with ZenFlow. (#987)
Signed-off-by: Tingfeng Lan <erc8gx@virginia.edu>
1 parent b4385e5 commit 01f520e

1 file changed

Lines changed: 3 additions & 3 deletions

File tree

  • training/DeepSpeed-ZenFlow/finetuning

training/DeepSpeed-ZenFlow/finetuning/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ Edit `zf_config.json` to enable ZenFlow:
3737
bash finetune_llama.sh
3838
```
3939

40-
This runs LLaMA-2 fine-tuning using DeepSpeed + ZenFlow, saving checkpoints to `./alpaca_output`.
40+
This runs LLaMA-2 fine-tuning on Alpaca-52K using DeepSpeed + ZenFlow, saving checkpoints to `./alpaca_output`.
4141

4242
## Example Output
4343

@@ -59,7 +59,7 @@ Step 13, Loss: 0.2453, Time: 1061.80ms
5959
## Key Insight
6060
Steps like 5,6 and 7 are accumulation steps where ZenFlow overlaps part of the optimizer step in the background. These steps remain fast (~700ms).
6161

62-
Steps 8 performs the remaining part of optimizer step and updates parameters to the GPU (2–2.2s).
62+
Step 8 performs the remaining part of optimizer step and updates parameters to the GPU (2–2.2s).
6363

6464
Without ZenFlow, a full update would take nearly 4 seconds, and ZenFlow distributes half of this cost across earlier accumulation steps via asynchronous overlap.
6565

@@ -72,7 +72,7 @@ This demonstrates how ZenFlow hides much of the CPU offload cost, enabling near
7272

7373
## Citation
7474

75-
To cite DeepSpeed Chat, please cite our [arxiv report](https://arxiv.org/abs/2505.12242):
75+
To cite ZenFlow, please cite our [arxiv report](https://arxiv.org/abs/2505.12242):
7676

7777
```bib
7878
@misc{lan2025zenflowenablingstallfreeoffloading,

0 commit comments

Comments
 (0)