MMLU Explained: What the 57-Subject LLM Benchmark Actually Tests
MMLU covers 57 academic subjects with 15,908 multiple-choice questions from elementary to professional level, and remains one of the most widely cited LLM benchmarks despite significant criticisms.
The Massive Multitask Language Understanding benchmark (arXiv:2009.03300) by Hendrycks et al. tests LLMs on 57 subject areas spanning STEM, humanities, social sciences, and professional domains. With 15,908 multiple-choice questions at difficulty levels from elementary to expert, it was designed to measure the breadth of knowledge a model has absorbed during pretraining.
Each question has four choices (A, B, C, D). The random baseline is 25%. MMLU is evaluated in two modes:
Few-shot (5-shot): Five example questions with answers are prepended to each test question. This tests whether the model can use the context format correctly and apply in-context learning.
Zero-shot: No examples. Tests pure parametric knowledge.
Score is reported as accuracy (fraction of questions answered correctly). Most papers use 5-shot.
Data contamination: Many MMLU questions appear verbatim in Common Crawl, meaning models may have memorized answers rather than demonstrating reasoning. Studies estimate contamination rates of 10-30% for some subjects.
Multiple-choice is not real-world: Production tasks require open-ended generation, not selecting from four options. A model can score 80% on MMLU while failing completely on open-ended versions of the same questions.
Inconsistent across papers: Different implementations (zero-shot vs few-shot, log-prob scoring vs generation, chain-of-thought or not) produce incomparable numbers. Papers cherry-pick the evaluation protocol that makes their model look best.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
SWE-Bench tests LLMs on 2,294 real GitHub issues from popular Python repositories, evaluating whether the model can write code that passes the existing test suite - a far harder and more realistic evaluation than HumanEval.
