Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Simplified MoE routing by steering each token to exactly one expert (Top-1 routing), reducing communication overhead and scaling models up to 1.6 trillion parameters.

Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-text-generation.

Assumptions

• Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support

Limitations

• Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology