Skip to main content

> ML_LITERATURE // SUN-2023-RETENTIVE-NETWORK-SUCCESSOR-TO-TRANSFORMER_v1.0

Retentive Network: A Successor to Transformer for Large Language Models (RetNet)

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyuan Wang, Furu Wei · arXiv preprint (2023)

seminal-architecture2023industry-standardthirdPartyReproduced
arXiv:2307.08621Back to Literature

Principal Contribution

Introduced the retention mechanism supporting three computation representations: parallel (for training), recurrent (for low-latency O(1) inference), and chunkwise (for long sequences).

Operational Relevance

Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-text-generation.

Assumptions

  • Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support

Limitations

  • Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology

Connected Algorithms, Architectures & Tools

Related Algorithms:
Related Architectures:
Implementing Libraries: