IndicIFEval
AI4BharatIndicIFEval tests whether language models follow instructions whose constraints a program can check, such as a word count, a required keyword or an output format, in 14 Indic languages. It has two parts. IndicIFEval-Trans adapts 490 prompts from the English IFEval benchmark to Indian contexts, machine-translates them, and has native speakers mark each translation correct or incorrect. IndicIFEval-Ground holds new prompts that Gemini 2.5 wrote from Indic web text, kept only where native speakers judged them clear and correct. AI4Bharat released it in 2026 with its evaluation harness.
The released translated prompts include those reviewers marked incorrect, tagged as such; the paper's scores use only the ones marked correct.
Openness
5 high confidence- license
- cc-by-4.0 (card metadata and API tag; the GitHub code, including the evaluation harness configs, is MIT)
- access
- public (ungated on the Hub)
- dataset_card
- present (card describes both subsets, the fields, the translation-quality tags and how to reproduce the paper's numbers, and links the paper and code)
The prompts download freely under CC BY 4.0, and the code that built and scores them is MIT. The card and the paper describe how each part was made and checked.
- https://huggingface.co/api/datasets/ai4bharat/IndicIFEval recorded 2026-11-17
API JSON: "gated":false; tag license:cc-by-4.0.
- https://huggingface.co/datasets/ai4bharat/IndicIFEval/raw/main/README.md recorded 2026-11-17
Card metadata "license: cc-by-4.0"; body: "IndicIFEval-Trans, translated prompts from IFEval carefully localized for Indic contexts, and IndicIFEval-Ground, synthetically generated instructions grounded in native Indic content".
- https://raw.githubusercontent.com/AI4Bharat/IndicIFEval/main/LICENSE recorded 2026-11-17
MIT License, Copyright (c) 2026 Thanmay Jayakumar.
- https://raw.githubusercontent.com/AI4Bharat/IndicIFEval/main/README.md recorded 2026-11-17
"This repository contains the complete codebase for the IndicIFEval benchmark from Data Creation to Benchmark Evaluation"; "This dataset is released under the CC BY 4.0".
Adoption
1 high confidenceHugging Face downloads of the Hub repository over the trailing month. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.
- https://huggingface.co/api/datasets/ai4bharat/IndicIFEval recorded 2026-11-17
"downloads":855 for ai4bharat/IndicIFEval.
Capability
3 medium confidenceIn each of its 14 target languages, IndicIFEval has between 440 and 490 translated prompts that native speakers marked correct, and those are the items that count toward depth. Its model-generated prompts were checked as well, but generated items do not count. An item count says how much a set can test, not how precisely it ranks models.
- https://arxiv.org/pdf/2602.22125 recorded 2026-11-17
"We use Sarvam-Translate (Sarvam, 2025) for the translations"; "In total, we arrive at 490 from the original 541 examples"; "We use Gemini-2.5 (Comanici et al., 2025) to generate a user prompt based on the provided context"; "we implement a human-in-the-loop verification process for both INDICIFEVAL-TRANS and INDICIFEVAL-GROUND subsets. We employ qualified, native speakers for each of the target languages as the annotators"; "For every prompt, annotators assessed three aspects"; Table 7, correct translations of 490: Assamese 440, Bengali 454, Gujarati 482, Hindi 487, Kannada 457, Malayalam 490, Marathi 483, Nepali 468, Odia 473, Punjabi 474, Sanskrit 450, Tamil 489, Telugu 488, Urdu 486; "our annotations were produced by a single annotator".
- https://datasets-server.huggingface.co/size?dataset=ai4bharat/IndicIFEval recorded 2026-11-17
The Hub's row counts, complete ("partial":false): indicifeval-trans 490 rows in each of 15 language splits including en; indicifeval-ground as 257, bn 409, gu 377, hi 457, kn 340, ml 384, mr 350, ne 341, or 366, pa 405, sa 336, ta 430, te 375, ur 337.
- https://huggingface.co/datasets/ai4bharat/IndicIFEval/raw/main/README.md recorded 2026-11-17
"For indicifeval-trans, every row carries exactly one of correct/incorrect (human translation-quality verdict)"; "To reproduce paper-reported numbers, filter to rows tagged correct"; IndicIFEval-Ground: "Synthetically generated instructions grounded in native Indic topics and content, manually verified by native speakers".
Verified 2026-11-17, except some scores, still awaiting re-confirmation