Back to Gap Map Model components / Language-specific evaluation sets

SEA-HELM

AI Singapore
gated / Overall score: 3.0

SEA-HELM (Southeast Asian Holistic Evaluation of Language Models) is AI Singapore's evaluation suite for large language models in Southeast Asian languages. It groups tasks under five pillars: classic NLP tasks, LLM-specific tasks such as instruction following and multi-turn chat, linguistic diagnostics, culture, and safety. Test sets combine native datasets, human translations and new handcrafted sets such as Kalahi. The suite absorbed the earlier BHASA benchmark and feeds the SEA-HELM leaderboard.

The suite's code lives on GitHub and its test sets on Hugging Face; many of those test sets are samples of existing public datasets. Kalahi's original repository, published before the suite renamed its task repositories, and SEA-HELM Audio, which reads two SEA-HELM tasks aloud in synthesized speech, are listed with the suite. Neither adds to the per-language depth below, since Kalahi's items are already counted and speech has no measurement model.

Openness

3 high confidence
3.0
license
mixed-per-subset (code is MIT; the README says all datasets are under their respective licenses (task repos range from cc-by-4.0 and cc-by-sa-4.0 to cc-by-nc-sa-4.0 and research-only sources))
access
manual (the most restrictive declared repository governs: Cultural-Evaluation-Kalahi-Judge needs a maintainer's approval, fifteen task repositories use Hugging Face's auto-approved gate with contact sharing, and kalahi and seahelm-audio are ungated)
dataset_card
present (each task card lists sources, licenses, languages and split statistics; docs/datasets_and_prompts.md covers the whole suite)

Most task data is available through automatically approved Hugging Face gates, but one task repository, the judged Kalahi set, needs a maintainer's approval, and the most restrictive gate is the one recorded. The evaluation code is MIT. Each task keeps the license of the dataset it samples, so some subsets are non-commercial or restricted to research use, including the Vietnamese hate-speech source.

Adoption

3 high confidence
3.0

Adoption is Hugging Face downloads summed over the eighteen task repositories. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations. Runs from local copies are not counted.

Capability

3 medium confidence
3.0

In the typical language SEA-HELM targets, its deepest test has four hundred to sixteen hundred items. Where that test comes from another dataset, as the FLORES translation test does in Khmer, Lao and Burmese, a separately built test in the same language backs it, so the corroborated count is the same. An item count says how much a set can test, not how precisely it ranks models.

Verified 2026-11-15