> ML_ARCHITECTURE // RWKV-RECURRENT-WEIGHTED-KEY-VALUE_v1.0
RWKV (Receptance Weighted Key Value Linear Hybrid)
Hybrid sequence architecture combining the parallelized training advantages of transformers with the constant-time and constant-memory inference efficiency of RNNs via linear attention formulations.
Linear Recurrent Modelstextcode
Back to All ArchitecturesArchitecture Overview
Hybrid sequence architecture combining the parallelized training advantages of transformers with the constant-time and constant-memory inference efficiency of RNNs via linear attention formulations.
Implementing Libraries
Seminal Papers
RWKV: Reinventing RNNs for the Transformer EraBo Peng, Eric Alcaide (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP)
Architectural Limitations & Constraints
- Requires compatible deep learning framework and hardware acceleration for efficient execution.