> ML_BENCHMARK // MMLU-MASSIVE-MULTITASK-LANGUAGE-UNDERSTANDING_v1.0
MMLU (Massive Multitask Language Understanding)
LLM General Knowledge & Reasoning · task-text-generation · near-saturation
LLM General Knowledge & Reasoningnear-saturationhigh-vram-dependent
Evaluation Protocol
5-shot multiple-choice question answering across 57 academic subjects (humanities, STEM, social sciences, business).
Baseline & Metrics
Canonical Baseline:Random Guess: 25.0% | GPT-4: 86.4% | Human Expert: ~89.8%
Evaluated Metrics:
5-shot Accuracy (%)
Contamination & Leakage Risks
High contamination risk in open-weights models trained on uncurated web crawls.
Reproducibility Concerns
Sensitive to prompt format, few-shot delimiter syntax, and choice logit normalization technique.