Skip to content

Commit

Permalink
refactor
Browse files Browse the repository at this point in the history
  • Loading branch information
@xrsrke
xrsrke committed Oct 24, 2023
1 parent bcf7531 commit 7b22bc1
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,9 @@ torchrun --standalone --nnodes=1 --nproc-per-node=4 hybrid_parallelism.py
```

We did a small scale correctness test by comparing the training losses between a paralleized transformer and one kept by default, starting at identical checkpoints and training data. We will conduct rigorous large scale convergence and weak scaling law benchmarks against Megatron and DeepSpeed in the near future.
- Data Parallelism [link](https://wandb.ai/xariusdrake/pipegoose/runs/smjfnm9g)
- Tensor Parallelism [link](https://wandb.ai/xariusdrake/pipegoose/runs/iz17f50n)
- Hybrid 2D Parallelism (TP+DP) [link](https://wandb.ai/xariusdrake/pipegoose/runs/us31p3q1)
- Data Parallelism [[link]](https://wandb.ai/xariusdrake/pipegoose/runs/smjfnm9g)
- Tensor Parallelism [[link]](https://wandb.ai/xariusdrake/pipegoose/runs/iz17f50n)
- Hybrid 2D Parallelism (TP+DP) [[link]](https://wandb.ai/xariusdrake/pipegoose/runs/us31p3q1)

**Features**
- Megatron-style 3D parallelism
Expand Down

0 comments on commit 7b22bc1

Please sign in to comment.