> ML_LITERATURE // ROZIERE-2023-CODE-LLAMA-OPEN-FOUNDATION-MODELS-CODE_v1.0
Code Llama: Open Foundation Models for Code
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Chintan Shah, Kamile Lukosuite, Thomas Hartvigsen, Arden Fallon, Alonso Pahuamba, Thomas Scialom, Gabriel Synnaeve · arXiv preprint (2023)
Principal Contribution
Specialised LLaMA 2 on 500B tokens of code with infilling capabilities (Fill-in-the-Middle FIM) and context window extension up to 100K tokens via modified RoPE frequencies.
Operational Relevance
Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-code-generation.
Assumptions
- Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support
Limitations
- Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology