> ML_LITERATURE // HU-2021-LORA-LOW-RANK-ADAPTATION-LARGE-LANGUAGE-MODELS_v1.0
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen · International Conference on Learning Representations (ICLR) (2021)
algorithm2021industry-standardthirdPartyReproduced
Principal Contribution
Decomposed dense weight update matrices into low-rank products (W + BA with rank r << d), reducing trainable parameters by 10,000x and GPU memory requirements by 3x.
Operational Relevance
Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-text-generation.
Assumptions
- Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support
Limitations
- Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology
Connected Algorithms, Architectures & Tools
Related Algorithms:
Related Architectures:
Implementing Libraries: