AI Potluck
Contribute
Back to Gap Map Model components / Benchmark / eval datasets

MMMU

Unknown
open / Overall score: 3.0

Massive Multi-discipline Multimodal Understanding: about 11,550 college-level questions combining images and text across 30 subjects and 183 subfields, using 30 image types, drawn from college exams, quizzes and textbooks. It splits into a 150-question dev set, a 900-question validation set and a 10,500-question test set.

Test-set answers were held out behind an evaluation server until 12 February 2026 and are now published, so every split is locally answerable.

Openness

5 high confidence
5.0
license
Apache-2.0(OSI/open,redistributable)
access
public(not gated)
datasheet
present(comprehensive card)
splits
dev+validation+test all public (test answers released 2026-02-12

Fully open: Apache-2.0, ungated, redistributable, comprehensive card. Test-set answers were released 2026-02-12 (previously held out via EvalAI), so there is no longer a held-out element, and the benchmark is fully public.

  • https://arxiv.org/abs/2311.16502 recorded 2026-08-13

    the abstract still describes 11.5K multimodal questions across six core disciplines from college exams, quizzes and textbooks

  • https://huggingface.co/datasets/MMMU/MMMU recorded 2026-10-13

    `license:apache-2.0` in the repo tags; the embedded repo state reads `"gated":false`; the card news list still reads "[2026-02-12]: We have released the answers for the test set!" and the EvalAI submission paragraph is struck through; dev/validation/test splits all present; 51,175 downloads in the trailing 30 days

Adoption

3 high confidence
3.0

Adoption is measured through Hugging Face downloads of MMMU/MMMU. MMMU and its MMMU-Pro extension are standard multimodal-understanding benchmarks, cited routinely in frontier multimodal model reports, though this category counts dataset downloads on a far smaller scale than it does for software or models.

Capability

not assessed

A dataset is not 'capable', so capability is left unscored; openness and adoption carry this category.

  • https://huggingface.co/datasets/MMMU/MMMU recorded 2026-09-29

    Dataset card describing a static question corpus: "The dataset contains 11.5K college-level problems across six broad disciplines" and "MMMU was created to challenge multimodal models with tasks that demand college-level subject knowledge"; the card embeds a "Mini-Leaderboard" whose rows are model and human-expert scores on the benchmark, not a capability of the corpus; searched the body for throughput, latency and SOTA: none.

Verified 2026-09-27