> ML_LITERATURE // SHOEYBI-2019-MEGATRON-LM-TRAINING-MULTI-BILLION-PARAMETER-LANGUAGE-MODELS_v1.0
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro · arXiv preprint (2019)
systems2019industry-standardartifactsAvailable
Principal Contribution
Introduced tensor parallel matrix multiplication across GPUs, splitting attention heads and MLP projections with minimal communication.
Operational Relevance
Directly guides deployment choices and architecture selection for task-text-generation.
Assumptions
- Standard empirical regularity and statistical stability hold across evaluation domains
Limitations
- Performance characteristics depend on domain distribution and compute allocation parameters
Connected Algorithms, Architectures & Tools
Related Algorithms:
Related Architectures:
Implementing Libraries: