GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Created automated compiler annotations for SPDM tensor sharding across thousands of TPU cores and top-2 gating with capacity limits for Sparsely-Gated MoE.

Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-text-generation, task-machine-translation.

Assumptions

• Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support

Limitations

• Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology