Improving Language Understanding by Generative Pre-Training (GPT-1)

Established generative autoregressive pre-training on diverse unlabeled text followed by discriminative task fine-tuning.

Origin of the GPT series, validating causal autoregressive Transformer decoding for general NLP task transfer.

Assumptions

• Unsupervised language modeling serves as an effective pre-training objective that transfers to diverse supervised downstream tasks

Limitations

• Model scale (117M parameters) was insufficient for zero-shot in-context task execution without fine-tuning