> ML_LITERATURE // LOZHKOV-2024-STARCODER-2-THE-STACK-V2_v1.0
StarCoder 2 and The Stack v2: The Next Generation
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Liu, Carolyn Jane Anderson, Luka Denouveaux, Stas Bekman, Armel Zebaze, Binyuan Hui, Jian Yang, Sadegh Farhadkhani, Guoqing Zheng, Zhiruo Wang, Suhas Kotha, Maxime Daisy, Hao Ding, Leon Shen, Urvashi Bhattacharyya, Quentin Anthony, Leandro von Werra, Harm de Vries · arXiv preprint (2024)
seminal-architecture2024industry-standardthirdPartyReproduced
Principal Contribution
Constructed The Stack v2 (over 600 programming languages) with rigorous attribution and opt-out mechanisms, releasing open models trained up to 4T tokens with Grouped Query Attention.
Operational Relevance
Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-code-generation.
Assumptions
- Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support
Limitations
- Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology
Connected Algorithms, Architectures & Tools
Related Algorithms:
Related Architectures:
Implementing Libraries: