SEA-HELM
AI SingaporeSEA-HELM (Southeast Asian Holistic Evaluation of Language Models) is AI Singapore's evaluation suite for large language models in Southeast Asian languages. It groups tasks under five pillars: classic NLP tasks, LLM-specific tasks such as instruction following and multi-turn chat, linguistic diagnostics, culture, and safety. Test sets combine native datasets, human translations and new handcrafted sets such as Kalahi. The suite absorbed the earlier BHASA benchmark and feeds the SEA-HELM leaderboard.
The suite's code lives on GitHub and its test sets on Hugging Face; many of those test sets are samples of existing public datasets. Kalahi's original repository, published before the suite renamed its task repositories, and SEA-HELM Audio, which reads two SEA-HELM tasks aloud in synthesized speech, are listed with the suite. Neither adds to the per-language depth below, since Kalahi's items are already counted and speech has no measurement model.
Openness
3 high confidence- license
- mixed-per-subset (code is MIT; the README says all datasets are under their respective licenses (task repos range from cc-by-4.0 and cc-by-sa-4.0 to cc-by-nc-sa-4.0 and research-only sources))
- access
- manual (the most restrictive declared repository governs: Cultural-Evaluation-Kalahi-Judge needs a maintainer's approval, fifteen task repositories use Hugging Face's auto-approved gate with contact sharing, and kalahi and seahelm-audio are ungated)
- dataset_card
- present (each task card lists sources, licenses, languages and split statistics; docs/datasets_and_prompts.md covers the whole suite)
Most task data is available through automatically approved Hugging Face gates, but one task repository, the judged Kalahi set, needs a maintainer's approval, and the most restrictive gate is the one recorded. The evaluation code is MIT. Each task keeps the license of the dataset it samples, so some subsets are non-commercial or restricted to research use, including the Vietnamese hate-speech source.
- https://huggingface.co/api/datasets?author=aisingapore&limit=200 recorded 2026-11-18
The Hub listing of AI Singapore's datasets gives "gated":"auto" for fifteen of the suite's declared task repositories, "gated":"manual" for Cultural-Evaluation-Kalahi-Judge and "gated":false for kalahi and seahelm-audio, with license tags from cc-by-4.0 and cc-by-sa-4.0 to cc-by-nc-sa-4.0, and none on NLU-Belebele-MCQA.
- https://huggingface.co/api/datasets/aisingapore/Cultural-Evaluation-Kalahi-Judge recorded 2026-11-18
API JSON: "gated":"manual"; tag license:cc-by-4.0.
- https://huggingface.co/collections/aisingapore/sea-helm-evaluation-datasets recorded 2026-09-24
Collection 'SEA-HELM Evaluation Datasets' listing the 14 task repositories
- https://huggingface.co/datasets/aisingapore/NLG-Machine-Translation recorded 2026-09-24
"You need to agree to share your contact information to access this dataset"; "It is part of the SEA-HELM leaderboard from AI Singapore"
- https://raw.githubusercontent.com/aisingapore/SEA-HELM/main/docs/datasets_and_prompts.md recorded 2026-11-18
Per-task tables of dataset, nativeness, domain and license, e.g. ViHSD "Research purposes only", XL-Sum "CC BY-NC-SA 4.0"
- https://raw.githubusercontent.com/aisingapore/SEA-HELM/main/LICENSE recorded 2026-11-18
MIT License, Copyright (c) 2025 AI Singapore
- https://raw.githubusercontent.com/aisingapore/SEA-HELM/main/README.md recorded 2026-11-18
"The codes for SEA-HELM is licensed under the MIT license. All datasets are licensed under their respective licenses."
Adoption
3 high confidenceAdoption is Hugging Face downloads summed over the eighteen task repositories. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations. Runs from local copies are not counted.
- https://huggingface.co/api/datasets/aisingapore/Cultural-Evaluation-Kalahi recorded 2026-09-24
947 downloads in the trailing 30 days for aisingapore/Cultural-Evaluation-Kalahi
- https://huggingface.co/api/datasets/aisingapore/Cultural-Evaluation-Kalahi-Judge recorded 2026-09-24
888 downloads in the trailing 30 days for aisingapore/Cultural-Evaluation-Kalahi-Judge
- https://huggingface.co/api/datasets/aisingapore/Instruction-Following-IFEval recorded 2026-09-24
2904 downloads in the trailing 30 days for aisingapore/Instruction-Following-IFEval
- https://huggingface.co/api/datasets/aisingapore/kalahi recorded 2026-11-17
44 downloads in the trailing 30 days for aisingapore/kalahi
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Pragmatics recorded 2026-09-24
1438 downloads in the trailing 30 days for aisingapore/Linguistic-Diagnostics-Pragmatics
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Syntax recorded 2026-09-24
1398 downloads in the trailing 30 days for aisingapore/Linguistic-Diagnostics-Syntax
- https://huggingface.co/api/datasets/aisingapore/MultiTurn-Chat-MT-Bench recorded 2026-09-24
80 downloads in the trailing 30 days for aisingapore/MultiTurn-Chat-MT-Bench
- https://huggingface.co/api/datasets/aisingapore/MultiTurn-Chat-MT-Bench-Judge recorded 2026-09-24
2270 downloads in the trailing 30 days for aisingapore/MultiTurn-Chat-MT-Bench-Judge
- https://huggingface.co/api/datasets/aisingapore/NLG-Abstractive-Summarization recorded 2026-09-24
2313 downloads in the trailing 30 days for aisingapore/NLG-Abstractive-Summarization
- https://huggingface.co/api/datasets/aisingapore/NLG-Machine-Translation recorded 2026-09-24
2893 downloads in the trailing 30 days for aisingapore/NLG-Machine-Translation
- https://huggingface.co/api/datasets/aisingapore/NLR-Causal-Reasoning recorded 2026-09-24
2428 downloads in the trailing 30 days for aisingapore/NLR-Causal-Reasoning
- https://huggingface.co/api/datasets/aisingapore/NLR-NLI recorded 2026-09-24
2399 downloads in the trailing 30 days for aisingapore/NLR-NLI
- https://huggingface.co/api/datasets/aisingapore/NLU-Belebele-MCQA recorded 2026-09-24
1588 downloads in the trailing 30 days for aisingapore/NLU-Belebele-MCQA
- https://huggingface.co/api/datasets/aisingapore/NLU-Metaphor recorded 2026-09-24
921 downloads in the trailing 30 days for aisingapore/NLU-Metaphor
- https://huggingface.co/api/datasets/aisingapore/NLU-Question-Answering recorded 2026-09-24
1860 downloads in the trailing 30 days for aisingapore/NLU-Question-Answering
- https://huggingface.co/api/datasets/aisingapore/NLU-Sentiment-Analysis recorded 2026-09-24
2599 downloads in the trailing 30 days for aisingapore/NLU-Sentiment-Analysis
- https://huggingface.co/api/datasets/aisingapore/Safety-Toxicity-Detection recorded 2026-09-24
2568 downloads in the trailing 30 days for aisingapore/Safety-Toxicity-Detection
- https://huggingface.co/api/datasets/aisingapore/seahelm-audio recorded 2026-11-17
344 downloads in the trailing 30 days for aisingapore/seahelm-audio
Capability
3 medium confidenceIn the typical language SEA-HELM targets, its deepest test has four hundred to sixteen hundred items. Where that test comes from another dataset, as the FLORES translation test does in Khmer, Lao and Burmese, a separately built test in the same language backs it, so the corroborated count is the same. An item count says how much a set can test, not how precisely it ranks models.
- https://aclanthology.org/2023.eacl-main.57.pdf recorded 2026-11-08
"we take an existing dataset in a high-resource local language (Indonesian) as a base for expansion to the other ten languages, and ask human annotators to translate the text."
- https://arxiv.org/abs/2207.04672 recorded 2026-11-08
"we evaluated the performance of over 40,000 different translation directions using a human-translated benchmark, Flores-200"
- https://arxiv.org/abs/2502.14301 recorded 2026-11-08
arXiv 2502.14301, SEA-HELM: Southeast Asian Holistic Evaluation of Language Models
- https://arxiv.org/html/2308.16884 recorded 2026-11-18
"Belebele was created end-to-end without the use of machine translation technology, relying solely on experts fluent in English and the target language"; "the context passages were taken directly from the FLoRes -200 dataset"
- https://arxiv.org/html/2502.14301 recorded 2026-11-08
Table 1 marks each dataset native or translated: NusaX, Wisesight, UIT-VSFC, TyDi QA-GoldP, IndicQA, MABL, IndoNLI, XL-Sum, LINDSEA, KALAHI, MLHSD, Thai Toxicity Tweet and ViHSD native; IndicSentiment, XQUAD, XNLI, XCOPA and MT-Bench Thai "Human" translation; IndicXNLI "Machine"; its caption: "We curate datasets that are created in the native language (indicated with "-") and translated datasets (either by humans or machines)"; "We selected datasets that comprised of data originally written in the native language as far as possible"; LINDSEA examples "are handcrafted from scratch by linguists in collaboration with native speakers"; Kalahi is "composed of 150 high-quality Filipino-language" "prompts created in collaboration with native Filipino speakers"
- https://huggingface.co/api/datasets/aisingapore/Instruction-Following-IFEval recorded 2026-11-17
the card's dataset_info: 105 "eval" examples in each of "km", "lo", "ms" and "my", languages the README's prose does not name
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Pragmatics recorded 2026-11-17
the card's dataset_info: 80 "eval" examples in "ta_single" and 84 in "ta_pair"
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Syntax recorded 2026-11-17
the card's dataset_info: 470 "eval" examples in "ta"
- https://huggingface.co/api/datasets/aisingapore/MultiTurn-Chat-MT-Bench recorded 2026-11-17
the card's dataset_info: 58 "eval" examples in each of "km", "lo", "ms", "my" and "tl"
- https://huggingface.co/api/datasets/aisingapore/NLG-Abstractive-Summarization recorded 2026-11-17
the card's dataset_info: 100 "eval" examples in "my"
- https://huggingface.co/api/datasets/aisingapore/NLG-Machine-Translation recorded 2026-11-08
the card's dataset_info, which the data files follow: "en_to_my" and "my_to_en" 600 eval examples each, "en_to_tl" and "tl_to_en" 600, "en_to_lo" and "lo_to_en" 1012; the README's table lists Burmese at 1012 and has no Lao or Filipino split
- https://huggingface.co/api/datasets/aisingapore/NLR-Causal-Reasoning recorded 2026-11-17
the card's dataset_info: 500 "eval" examples in "ms" and 400 in "my"
- https://huggingface.co/api/datasets/aisingapore/NLR-NLI recorded 2026-11-17
the card's dataset_info: 1000 "eval" examples in "ms" and 600 in "my"
- https://huggingface.co/api/datasets/aisingapore/NLU-Belebele-MCQA recorded 2026-11-17
the card's dataset_info, which the data files follow: "eval" splits of 895 examples for "id", "jv", "km", "lo", "ms", "su", "ta", "th" and "vi", 120 for "my"; the README's prose names none of them
- https://huggingface.co/datasets/aisingapore/Cultural-Evaluation-Kalahi recorded 2026-11-08
"Kalahi only has a Tagalog (tl) split."; 150 examples in "tl"
- https://huggingface.co/datasets/aisingapore/Cultural-Evaluation-Kalahi-Judge recorded 2026-11-08
"Kalahi only has a Tagalog (tl) split."; 27 examples in "tl"
- https://huggingface.co/datasets/aisingapore/Instruction-Following-IFEval recorded 2026-11-08
"It is based on IFEval and was manually translated by native speakers for Indonesian, Javanese, Sundanese, Thai, Tagalog, and Vietnamese."; 105 examples in each of the "en", "id", "jv", "su", "th", "tl" and "vi" splits
- https://huggingface.co/datasets/aisingapore/Linguistic-Diagnostics-Pragmatics recorded 2026-11-08
"LINDSEA Pragmatics only has an Indonesian (id) split"; 100 examples in "id"
- https://huggingface.co/datasets/aisingapore/Linguistic-Diagnostics-Syntax recorded 2026-11-08
"LINDSEA Syntax only has an Indonesian (id) split"; 380 examples in "id"
- https://huggingface.co/datasets/aisingapore/MultiTurn-Chat-MT-Bench recorded 2026-11-08
"It is based on MT-Bench and was manually translated by native speakers for Indonesian (id), Javanese (jv), Sundanese (su), and Vietnamese (vi). The Thai split of this dataset uses MT-Bench Thai from the ThaiLLM leaderboard."; 58 examples in each of "en", "id", "jv", "su" and "vi", 91 in "th"
- https://huggingface.co/datasets/aisingapore/MultiTurn-Chat-MT-Bench-Judge recorded 2026-11-08
"The prompts are based on MT-Bench and was manually translated by native speakers."; 58 examples in each of "en", "id", "th", "vi", "ms", "tl" and "ta"
- https://huggingface.co/datasets/aisingapore/NLG-Abstractive-Summarization recorded 2026-11-08
"It is sampled from XL-Sum for Indonesian, Tamil, Thai, and Vietnamese."; 100 examples in each of the "id", "ta", "th" and "vi" splits
- https://huggingface.co/datasets/aisingapore/NLG-Machine-Translation recorded 2026-11-08
"It is sampled from FLORES 200 for Burmese, Chinese, English, Indonesian, Khmer, Malay, Tamil, Thai, and Vietnamese, and NusaX for Indonesian, Javanese, and Sundanese."; 1012 examples in each direction between English and ind_Latn, khm_Khmr, mya_Mymr, tam_Taml, tha_Thai, vie_Latn, zho_Hans and zsm_Latn; 394 in each of "id_to_jv", "jv_to_id", "id_to_su" and "su_to_id"
- https://huggingface.co/datasets/aisingapore/NLR-Causal-Reasoning recorded 2026-11-08
"It is sampled from XCOPA for Indonesian, Tamil, Thai, and Vietnamese."; 500 examples in each of the "id", "ta", "th" and "vi" splits
- https://huggingface.co/datasets/aisingapore/NLR-NLI recorded 2026-11-08
"It is sampled from IndoNLI for Indonesian, IndicXNLI for Tamil, and XNLI for Thai and Vietnamese."; 1000 examples in each of the "id", "ta", "th" and "vi" splits (the card is headed "SEA Abstractive Summarization", a copy error)
- https://huggingface.co/datasets/aisingapore/NLU-Metaphor recorded 2026-11-08
"It is sampled from Multilingual-Fig-QA for Indonesian, Javanese, and Sundanese."; split sizes "id" 295, "jv" 282, "su" 295
- https://huggingface.co/datasets/aisingapore/NLU-Question-Answering recorded 2026-11-08
"It is sampled from TyDi QA-GoldP for Indonesian, IndicQA for Tamil, and XQuaD for Thai and Vietnamese."; 100 examples in each of the "id", "ta", "th" and "vi" splits
- https://huggingface.co/datasets/aisingapore/NLU-Sentiment-Analysis recorded 2026-11-08
"It is sampled from NusaX for Indonesian, Javanese, and Sundanese, IndicSentiment for Tamil, Wisesight Sentiment for Thai, and UIT-VSFC for Vietnamese."; split sizes "id" 400, "jv" 394, "su" 394, "ta" 1000, "th" 1000, "vi" 1000, plus five few-shot examples per language
- https://huggingface.co/datasets/aisingapore/Safety-Toxicity-Detection recorded 2026-11-08
"It is sampled from MLHSD for Indonesian, TTD for Thai, and ViHSD for Vietnamese."; 1000 examples in each of the "id", "th" and "vi" splits
- https://raw.githubusercontent.com/aisingapore/SEA-HELM/main/docs/datasets_and_prompts.md recorded 2026-11-17
Task tables name FLORES, XL-Sum, NusaX, XNLI, Belebele, Kalahi, LINDSEA, SEA-IFEval, SEA MT-Bench, Thai Exam; the sentiment table marks "Javanese" and "Sudanese" "Native" with no dataset named, and the metaphor table marks "Javanese, Sudanese" "Native"; the translation table reads "Indonesian, Tamil, Thai, Vietnamese, Lao, Malay, Burmese | FLORES | Human translations" and "Filipino | Batayan | Human translations | Wikinews, Wikijunior, and Wikivoyage"; the tables also name the languages added later, in "Filipino, Indonesian, Vietnamese, Malay, Burmese, Lao, Khmer | SEA-IFEval | Human translation", "Filipino, Indonesian, Thai, Vietnamese, Malay, Burmese, Lao, Khmer | SEA MT-Bench | Human Translation", "Malay, Burmese, Lao, Javanese, Sundanese | Belebele | Human translation", "Thai, Vietnamese, Malay, Burmese | XNLI | Human translation", "Indonesian, Tamil, Thai, Vietnamese, Malay, Burmese | XCOPA | Human translation", "Indonesian, Tamil, Thai, Vietnamese, Burmese | XL-Sum | Native" and "Indonesian, Tamil | LINDSEA | Native"; the sentiment table marks "Malay", "Burmese" and "Lao" "Native" and the toxicity table "Burmese" and "Malay" "Native", each with no dataset named
- https://raw.githubusercontent.com/aisingapore/SEA-HELM/main/README.md recorded 2026-11-08
"This suite consist of 5 core pillars: NLP Classics, LLM-specifics, SEA Linguistics, SEA Culture, Safety"; "BHASA has been integrated into SEA-HELM"
- https://raw.githubusercontent.com/aisingapore/SEA-HELM/main/seahelm_tasks/nlu/belebele_mcqa/config.yaml recorded 2026-11-17
the belebele-qa-mc task maps "id", "jv", "km", "lo", "my", "su", "ta", "th" and "vi" each to "filepath: aisingapore/NLU-Belebele-MCQA"
Verified 2026-11-15