> ML_BENCHMARK // TRUTHFULQA-BENCHMARK_v1.0
TruthfulQA (Model Falsehood & Hallucination Benchmark)
Safety & Epistemic Truthfulness · task-question-answering · active-headroom
Safety & Epistemic Truthfulnessactive-headroomdeterministic-cpu
Evaluation Protocol
817 questions in zero-shot setting evaluated across multiple-choice selection and free-form generation judged by fine-tuned truthfulness classifier.
Baseline & Metrics
Canonical Baseline:GPT-3 (original): 28.0% | Llama 2 70B: 50.2% | Human: ~94.0%
Evaluated Metrics:
% Truthful * % Informative (MC1, MC2, Fine-tuned GPT Judge)
Contamination & Leakage Risks
Fine-tuning directly on TruthfulQA can artificially inflate scores without resolving core hallucinations.
Reproducibility Concerns
GPT-Judge fine-tuned model versions must match exactly across evaluation runs for cross-comparison.
CONNECTED DATASETTruthfulQA: Measuring Model Imitation of FalsehoodsEpistemic Truthfulness & Alignment · 817 adversarial questions across 38 categories (health, law, conspiracies, superstitions)
View Dataset Spec