Resources for large scale models

Related to data parallelism, model parallelism, pipeline parallelism and their variants

Model Parallelismhuggingface

dlatscale_draft/week03 at main · yandexdataschool/dlatscale_draftGitHub

Mixture of Experts

FastMOE: https://arxiv.org/abs/2103.13262, https://github.com/laekov/fastmoe