LINDSEA
AI SingaporeLINDSEA is a linguistic diagnostic for language models in Indonesian and Tamil. Linguists and native speakers wrote its items from scratch to probe syntax, such as agreement and word order, and pragmatics, such as presupposition and implicature. AI Singapore introduced it in the BHASA paper, and it now runs as SEA-HELM's linguistic diagnostics, published as two Hub repositories.
Openness
3 high confidence- license
- cc-by-4.0 (card metadata and API tag on both repositories)
- access
- auto (Hugging Face gate that approves anyone who agrees to share their contact information, on both repositories)
- dataset_card
- present (cards name the BHASA source, license, languages and split statistics)
Both parts carry CC BY 4.0, and the Hugging Face copies need an access request that is approved automatically. The BHASA paper describes how the items were written.
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Pragmatics recorded 2026-11-18
API JSON: "gated":"auto"; tag license:cc-by-4.0.
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Syntax recorded 2026-11-18
API JSON: "gated":"auto"; tag license:cc-by-4.0.
- https://huggingface.co/datasets/aisingapore/Linguistic-Diagnostics-Pragmatics recorded 2026-11-18
Card: "LINDSEA Pragmatics is a linguistic diagnostic from BHASA that evaluates a model's understanding of linguistic phenomena, pragmatics in particular"; a data-source table naming BHASA and CC BY 4.0.
- https://huggingface.co/datasets/aisingapore/Linguistic-Diagnostics-Syntax recorded 2026-11-18
Card: "LINDSEA Syntax is a linguistic diagnostic from BHASA that evaluates a model's understanding of linguistic phenomena, syntax in particular"; a data-source table naming BHASA and CC BY 4.0.
Adoption
2 high confidenceHugging Face downloads summed over the syntax and pragmatics repositories over the trailing month. Evaluation harnesses, SEA-HELM's among them, can fetch a test set repeatedly, so the figure is a rough measure of use, not a count of users.
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Pragmatics recorded 2026-11-18
"downloads":1300 for aisingapore/Linguistic-Diagnostics-Pragmatics.
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Syntax recorded 2026-11-18
"downloads":1597 for aisingapore/Linguistic-Diagnostics-Syntax.
Capability
2 medium confidenceLINDSEA's deepest test, syntax, has four hundred to sixteen hundred items in Tamil and one to four hundred in Indonesian, all written for the set by linguists and native speakers. An item count says how much a set can test, not how precisely it ranks models.
- https://arxiv.org/abs/2309.06085 recorded 2026-11-18
Abstract: BHASA "comprises three components", among them "LINDSEA, a linguistic diagnostic toolkit that spans the gamut of linguistic phenomena including syntax, semantics and pragmatics".
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Pragmatics recorded 2026-11-18
the card's dataset_info: 100 "eval" examples in "id_single", 84 in "id_pair", 80 in "ta_single" and 84 in "ta_pair"
- https://huggingface.co/api/datasets/aisingapore/Linguistic-Diagnostics-Syntax recorded 2026-11-18
the card's dataset_info: 380 "eval" examples in "id" and 470 in "ta"
- https://raw.githubusercontent.com/aisingapore/SEA-HELM/main/docs/datasets_and_prompts.md recorded 2026-11-18
"Indonesian, Tamil | LINDSEA | Native"
Verified 2026-11-18, except some scores, still awaiting re-confirmation