Skip to main content

> ML_LITERATURE // RAJBHANDARI-2020-ZERO-MEMORY-OPTIMIZATIONS-TOWARD-TRAINING-TRILLION-PARAMETER-MODELS_v1.0

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He · IEEE/ACM International Conference on High Performance Computing (SC20) (2020)

systems2020industry-standardartifactsAvailable
DOI: 10.1109/SC41405.2020.00024Back to Literature

Principal Contribution

Introduced the Zero Redundancy Optimizer (ZeRO Stages 1-3), partitioning optimizer states, gradients, and model parameters across data-parallel ranks.

Operational Relevance

Directly guides deployment choices and architecture selection for task-text-generation.

Assumptions

  • Standard empirical regularity and statistical stability hold across evaluation domains

Limitations

  • Performance characteristics depend on domain distribution and compute allocation parameters

Connected Algorithms, Architectures & Tools

Related Algorithms:
Related Architectures:
Implementing Libraries: