Back to Gap Map Model components / Language-specific evaluation sets

LINDSEA

AI Singapore
gated / Overall score: 2.0

LINDSEA is a linguistic diagnostic for language models in Indonesian and Tamil. Linguists and native speakers wrote its items from scratch to probe syntax, such as agreement and word order, and pragmatics, such as presupposition and implicature. AI Singapore introduced it in the BHASA paper, and it now runs as SEA-HELM's linguistic diagnostics, published as two Hub repositories.

Openness

3 high confidence
3.0
license
cc-by-4.0 (card metadata and API tag on both repositories)
access
auto (Hugging Face gate that approves anyone who agrees to share their contact information, on both repositories)
dataset_card
present (cards name the BHASA source, license, languages and split statistics)

Both parts carry CC BY 4.0, and the Hugging Face copies need an access request that is approved automatically. The BHASA paper describes how the items were written.

Adoption

2 high confidence
2.0

Hugging Face downloads summed over the syntax and pragmatics repositories over the trailing month. Evaluation harnesses, SEA-HELM's among them, can fetch a test set repeatedly, so the figure is a rough measure of use, not a count of users.

Capability

2 medium confidence
2.0

LINDSEA's deepest test, syntax, has four hundred to sixteen hundred items in Tamil and one to four hundred in Indonesian, all written for the set by linguists and native speakers. An item count says how much a set can test, not how precisely it ranks models.

Verified 2026-11-18, except some scores, still awaiting re-confirmation