Language Models are Unsupervised Multitask Learners (GPT-2)

Demonstrated that scaling autoregressive language models (1.5B parameters) to diverse web datasets unlocks zero-shot task completion without fine-tuning.

Proved the concept of prompt-based zero-shot evaluation, shifting AI away from task-specific fine-tuning heads.

Assumptions

• Language models trained on sufficiently large text corpora begin learning to solve downstream tasks conditioned solely on natural language prompts

Limitations

• Hallucination, factual inconsistency, and sensitivity to prompt formulation; lack of safety alignment