Skip to main content

> ML_LITERATURE // VASWANI-2017-ATTENTION-IS-ALL-YOU-NEED_v1.0

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin · Advances in Neural Information Processing Systems (NeurIPS) (2017)

seminal-architecture2017foundationalthirdPartyReproduced
arXiv:1706.03762Back to Literature

Principal Contribution

Replaced recurrence and convolutions entirely with multi-head self-attention, inventing the Transformer architecture.

Operational Relevance

The universal base architecture for modern AI: GPT, Claude, Gemini, LLaMA, ViT, Whisper, and AlphaFold.

Assumptions

  • Pairwise token interactions can model all contextual linguistic dependencies without sequential recurrence constraints

Limitations

  • Self-attention memory and compute scale quadratically O(N^2) with sequence length N

Connected Algorithms, Architectures & Tools

Related Algorithms:
alg-autoregressive-self-supervised-gptMasked Language Modeling (MLM / BERT)
Related Architectures:
Implementing Libraries: