> ML_LITERATURE // ZHENG-2023-JUDGING-LLM-AS-A-JUDGE-MT-BENCH-CHATBOT-ARENA_v1.0
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Hao Zhang, Siyuan Zhuang, Yonghao Zhuang, Lingxiao Zheng, Zhuohan Li, Zac Frady, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica · Advances in Neural Information Processing Systems (NeurIPS) (2023)
benchmark2023foundationalartifactsAvailable
Principal Contribution
Created the crowdsourced blind A/B Chatbot Arena and MT-Bench evaluation protocol with LLM-as-a-Judge, adopting Bradley-Terry Elo ratings for language model evaluation.
Operational Relevance
Serves as qualified theoretical and empirical reference for task-text-generation, task-model-evaluation.
Assumptions
- Mathematical convexity, regularity, and empirical consistency hold across problem configurations
Limitations
- Theoretical bounds and benchmark saturation characteristics vary across model architectures and training scales
Connected Algorithms, Architectures & Tools
Related Algorithms:
Related Architectures:
Implementing Libraries: