> ML_BENCHMARK // LMSYS-CHATBOT-ARENA-ELO_v1.0
LMSYS Chatbot Arena (Crowdsourced Human Preference Elo)
Chat & Alignment · task-text-generation · active-headroom
Chat & Alignmentactive-headroommoderate
Evaluation Protocol
Blind side-by-side human A/B testing on arbitrary user prompts with randomized model pairings and statistical bootstrapping.
Baseline & Metrics
Canonical Baseline:GPT-4o: ~1330 Elo | Claude 3.5 Sonnet: ~1335 Elo | Gemini 1.5 Pro: ~1300 Elo
Evaluated Metrics:
Bradley-Terry Arena Elo Rating
Contamination & Leakage Risks
Near zero contamination because queries originate dynamically from live global human users.
Reproducibility Concerns
Dependent on live human voter distribution, prompt length bias, and prompt style variations.
CONNECTED DATASETLMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation DatasetConversational AI & User Intent · 1 million real-world user conversations with 25 frontier LLMs collected from Chatbot Arena
View Dataset Spec