Scaling Laws for Neural Language Models

Empirically demonstrated that language model cross-entropy loss scales as a power-law with model size N, dataset size D, and compute budget C over several orders of magnitude.

Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-text-generation.

Assumptions

• Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support

Limitations

• Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology