Back to Gap Map Model components / Language-specific evaluation sets

SEA-MTBench

AI Singapore
gated / Overall score: 1.0

SEA-MTBench tests how language models hold a two-turn conversation in Southeast Asian languages, judged by a stronger model against a fixed baseline. AI Singapore translated and localized the English MT-Bench questions by hand, 58 per language, and uses the 91 questions of the ThaiLLM leaderboard's MT-Bench Thai for Thai. It is SEA-HELM's multi-turn chat task and is published on its own on the Hub.

Openness

3 low confidence
3.0
license
cc-by-4.0 (card metadata and API tag, for the translated splits)+apache-2.0 (card metadata and API tag, for the Thai split from MT-Bench Thai)
access
auto (Hugging Face gate that approves anyone who agrees to share their contact information)
dataset_card
present (card lists the sources, licenses, languages and split statistics)

The Hub card and tags give CC BY 4.0 for the translated questions and Apache 2.0 for the Thai split, and the copy needs an access request that is approved automatically. SEA-HELM's own dataset tables give CC BY-NC-SA 4.0 for SEA MT-Bench, so the two first-party statements disagree and the reading is held.

Adoption

1 high confidence
1.0

Hugging Face downloads of the repository over the trailing month. SEA-HELM declares the same repository, so these downloads also count toward SEA-HELM's figure, and the runs that read its separate judged copy count only there.

Capability

1 medium confidence
1.0

In every language SEA-MTBench targets, its one test has fewer than a hundred questions, as MT-Bench itself does. Tamil has a split, but no source says how it was made, so it does not count. An item count says how much a set can test, not how precisely it ranks models.

Verified 2026-11-18, except some scores, still awaiting re-confirmation