> ML_LITERATURE // SU-2021-ROFORMER-ENHANCED-TRANSFORMER-WITH-ROTARY-POSITION-EMBEDDING-ROPE_v1.0
RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE)
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu · arXiv preprint (2021)
seminal-architecture2021industry-standardthirdPartyReproduced
Principal Contribution
Invented Rotary Position Embedding (RoPE), encoding relative positional information by rotating query and key vectors in complex 2D subspaces.
Operational Relevance
Serves as canonical technical reference for implementing task-text-generation, task-long-context-inference in production systems.
Assumptions
- Underlying data distribution satisfies empirical consistency and regularity assumptions across training domains
Limitations
- Scaling characteristics and accuracy depend on hardware architecture, parameter scale, and dataset quality
Connected Algorithms, Architectures & Tools
Related Algorithms:
Related Architectures:
Implementing Libraries: