MILU
AI4BharatMILU (Multi-task Indic Language Understanding Benchmark) is an evaluation set of about 80,000 multiple-choice questions in 10 Indic languages and English, spanning 8 domains and 41 subjects. Its questions come largely from regional and state examinations, about a quarter of them translated, and cover India-specific topics such as local history, arts and law. AI4Bharat publishes it with an lm-evaluation-harness setup for running it.
Openness
3 high confidence- license
- cc-by-4.0 (card metadata and README "License" section; the repository LICENSE.md is EleutherAI's MIT license for the harness code)
- access
- auto (Hugging Face gated: auto)
- dataset_card
- present (card gives per-language statistics, subjects and splits)
The benchmark carries CC BY 4.0, and the Hugging Face copy needs an access request that is approved automatically. The MIT license file in the GitHub repository belongs to the EleutherAI harness code, not the questions.
- https://huggingface.co/api/datasets/ai4bharat/MILU recorded 2026-09-24
"gated": "auto"; cardData license "cc-by-4.0".
- https://huggingface.co/datasets/ai4bharat/MILU recorded 2026-09-24
Per-language table: 79,617 questions, 19,915 translated; 8,933-sample validation set for few-shot use.
- https://raw.githubusercontent.com/AI4Bharat/MILU/master/LICENSE.md recorded 2026-09-24
"MIT License / Copyright (c) 2020 EleutherAI".
- https://raw.githubusercontent.com/AI4Bharat/MILU/master/README.md recorded 2026-09-24
README: "This dataset is released under the CC BY 4.0." Code evaluates models "using the lm-eval-harness framework".
Adoption
2 high confidenceHugging Face downloads of the MILU repository. Evaluation runs through the harness also fetch from here, but a download does not show which published results used it.
- https://huggingface.co/api/datasets/ai4bharat/MILU recorded 2026-09-24
1343 downloads in the trailing 30 days for ai4bharat/MILU
Capability
4 medium confidenceIn the typical language MILU targets, its deepest test has sixteen hundred to sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.
- https://arxiv.org/abs/2411.02538 recorded 2026-11-08
Abstract: "MILU spans 8 domains and 41 subjects across 11 Indic languages ... We evaluate over 42 LLMs".
- https://arxiv.org/pdf/2411.02538 recorded 2026-11-08
Table 2, total and translated questions per language: bn 6638 and 1601, gu 4827 and 2755, hi 14837 and 115, kn 6234 and 1522, ml 4321 and 3354, mr 6924 and 1235, or 4525 and 3100, pa 4099 and 3411, ta 6372 and 1524, te 7304 and 1298, en 13536 and none; "we sampled questions from the English set from that subject and translated them into the required language using GPT-4"; "only 25% of questions are translated from English, with the remainder"; "preserved in their original source languages"; "MILU is a large, multi-domain test set"
- https://huggingface.co/datasets/ai4bharat/MILU recorded 2026-11-08
the card's Dataset Statistics table gives the same per-language Total Questions and Translated Questions; "Use 'test' split for evaluation & 'validation' split for few-shot"; "a separate validation set of 8,933 samples that can be used for few-shot examples during evaluation", with no per-language count for it
Verified 2026-09-24