> ML_BENCHMARK // ARC-CHALLENGE-REASONING_v1.0
ARC Challenge (AI2 Reasoning Challenge 2590 Question Subset)
Commonsense & Scientific Reasoning · task-question-answering · saturated
Commonsense & Scientific Reasoningsaturatedmoderate
Evaluation Protocol
Multiple-choice question answering on questions that retrieval algorithms and co-occurrence baselines fail to answer correctly.
Baseline & Metrics
Canonical Baseline:Random: 25.0% | GPT-3.5: 85.0% | Claude 3.5 Sonnet: 96.7%
Evaluated Metrics:
25-shot Accuracy (%)
Contamination & Leakage Risks
Widespread memorization across open weights LLMs.
Reproducibility Concerns
Standard choice logit normalization technique.