Back to Gap Map Model components / Language-specific evaluation sets

SeaExam

Alibaba DAMO Academy
open / Overall score: 2.4

SeaExam is a multiple-choice benchmark from DAMO-NLP-SG, the group behind SeaLLMs, for testing language models in Indonesian, Thai, Vietnamese, Chinese and English. It combines real human exam questions adapted from M3Exam, standardized to four shuffled options, with 2,850 MMLU questions machine-translated using Google Translate. Its companion SeaBench adds open-ended multi-turn and instruction-following questions in Indonesian, Thai and Vietnamese, and both feed the SeaExam leaderboard.

Openness

5 high confidence
5.0
license
apache-2.0 (card metadata for both SeaExam and SeaBench)
access
public (both repos ungated)
dataset_card
present (describes the M3Exam adjustments and the MMLU translation; SeaBench card is brief)

Both sets download without a gate under Apache-2.0, answers included, and the SeaExam card explains how each part was built. The SeaBench card says little beyond its languages and purpose.

Adoption

1 high confidence
1.0

Adoption is Hugging Face downloads summed over SeaExam and SeaBench, most of them for SeaExam. Evaluations run through the project's own leaderboard code or local copies are not separately counted.

Capability

3 medium confidence
3.0

In the typical language SeaExam targets, its deepest test has four hundred to sixteen hundred items that count toward depth. In one of its 4 languages a test taken from another dataset counts for less, because nothing independent backs it there. An item count says how much a set can test, not how precisely it ranks models.

  • https://aclanthology.org/2025.findings-naacl.341.pdf recorded 2026-11-08

    "we decide to manually collect exam questions from the SEA region (Indonesian (id), Thai (th), and Vietnamese (vi))"; "we collaborate with native linguists from the SEA region to systematically collect official region-specific exam questions"; "The final SeaExam comprises a total of 5,451 test samples", Table 1 totals 1,580 (id), 2,126 (th) and 1,745 (vi); on MMLU, "Then we translate the questions and the choices from English into Indonesian, Thai, and Vietnamese using Google Translate API" and "We call the curated benchmark MMLU-SEA"

  • https://aclanthology.org/2025.findings-naacl.341/ recorded 2026-11-08

    SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia, Liu et al.

  • https://arxiv.org/pdf/2306.05179 recorded 2026-11-08

    M3Exam: "we select 9 languages including English (from the US), Chinese (from China)" and the rest, ending "and Javanese (from Indonesia)"; "We then engage native speakers from each of the selected countries to collect official exam papers along with their answers"; "collecting real-world natural data of different languages instead of translating from English data"; "separate these as held-out development data"; "The remaining questions are used as test data during the experiment"

  • https://datasets-server.huggingface.co/size?dataset=SeaLLMs/SeaExam recorded 2026-11-08

    the Hub's split sizes per config, test: "m3exam-english" 1908, "m3exam-chinese" 644, "m3exam-indonesian" 1580, "m3exam-thai" 2126, "m3exam-vietnamese" 1745, and 2850 in each of "mmlu-english", "mmlu-chinese", "mmlu-indonesian", "mmlu-thai" and "mmlu-vietnamese"; dev: 32, 25, 27, 26 and 25 in the same m3exam configs, and 285 in each mmlu config

  • https://huggingface.co/datasets/SeaLLMs/SeaExam recorded 2026-11-08

    Number of rows: 23,813; Space using SeaLLMs/SeaExam: SeaLLMs/LLM_Leaderboard_for_SEA

  • https://huggingface.co/datasets/SeaLLMs/SeaExam/raw/main/README.md recorded 2026-11-08

    "The original M3Exam dataset is constructed with real human exam questions"; "We randomly selected 50 questions from each subject, totaling 2850 questions"; under "Translated MMLU": "We translate the original English questions to different languages to measure the cross-lingual alignment" and "These questions have been translated from English into Chinese, Indonesian, Thai, and Vietnamese using Google Translate"; configs "m3exam-indonesian", "m3exam-thai", "m3exam-vietnamese", "m3exam-english", "m3exam-chinese" and "mmlu-english", "mmlu-chinese", "mmlu-thai", "mmlu-vietnamese", "mmlu-indonesian", each with a dev and a test split

  • https://raw.githubusercontent.com/DAMO-NLP-SG/SeaExam/main/README.md recorded 2026-11-08

    "The leaderboard showcases results from two complementary benchmarks: SeaExam and SeaBench"; cites Findings of NAACL 2025 paper

Verified 2026-09-24