Improving Language Understanding by Generative Pre-Training (GPT-1)

Introduced the two-stage training paradigm: generative autoregressive pretraining on diverse unlabeled text followed by supervised discriminative fine-tuning.

Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-text-generation, task-text-classification.

Assumptions

• Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support

Limitations

• Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology