Back to Gap Map Model components / Language-specific evaluation sets

MILU

AI4Bharat
gated / Overall score: 3.4

MILU (Multi-task Indic Language Understanding Benchmark) is an evaluation set of about 80,000 multiple-choice questions in 10 Indic languages and English, spanning 8 domains and 41 subjects. Its questions come largely from regional and state examinations, about a quarter of them translated, and cover India-specific topics such as local history, arts and law. AI4Bharat publishes it with an lm-evaluation-harness setup for running it.

Openness

3 high confidence
3.0
license
cc-by-4.0 (card metadata and README "License" section; the repository LICENSE.md is EleutherAI's MIT license for the harness code)
access
auto (Hugging Face gated: auto)
dataset_card
present (card gives per-language statistics, subjects and splits)

The benchmark carries CC BY 4.0, and the Hugging Face copy needs an access request that is approved automatically. The MIT license file in the GitHub repository belongs to the EleutherAI harness code, not the questions.

Adoption

2 high confidence
2.0

Hugging Face downloads of the MILU repository. Evaluation runs through the harness also fetch from here, but a download does not show which published results used it.

Capability

4 medium confidence
4.0

In the typical language MILU targets, its deepest test has sixteen hundred to sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.

  • https://arxiv.org/abs/2411.02538 recorded 2026-11-08

    Abstract: "MILU spans 8 domains and 41 subjects across 11 Indic languages ... We evaluate over 42 LLMs".

  • https://arxiv.org/pdf/2411.02538 recorded 2026-11-08

    Table 2, total and translated questions per language: bn 6638 and 1601, gu 4827 and 2755, hi 14837 and 115, kn 6234 and 1522, ml 4321 and 3354, mr 6924 and 1235, or 4525 and 3100, pa 4099 and 3411, ta 6372 and 1524, te 7304 and 1298, en 13536 and none; "we sampled questions from the English set from that subject and translated them into the required language using GPT-4"; "only 25% of questions are translated from English, with the remainder"; "preserved in their original source languages"; "MILU is a large, multi-domain test set"

  • https://huggingface.co/datasets/ai4bharat/MILU recorded 2026-11-08

    the card's Dataset Statistics table gives the same per-language Total Questions and Translated Questions; "Use 'test' split for evaluation & 'validation' split for few-shot"; "a separate validation set of 8,933 samples that can be used for few-shot examples during evaluation", with no per-language count for it

Verified 2026-09-24