Back to Gap Map Model components / Language-specific evaluation sets

IndicXTREME

AI4Bharat
restricted / Overall score: 3.4

IndicXTREME is a natural language understanding benchmark of nine tasks across 20 Indian languages, grouping 105 evaluation sets for classification, structure prediction, question answering and sentence retrieval. AI4Bharat created new sets, among them IndicCOPA, IndicQA, IndicXParaphrase and IndicSentiment, and folded in Naamapadam, IndicXNLI, MASSIVE and FLORES. It was released with IndicBERT v2 to test zero-shot multilingual transfer.

Listed are the AI4Bharat-hosted task repositories; the IndicXNLI, MASSIVE and FLORES tasks belong to other publishers and are not counted here.

Openness

2 medium confidence
2.0
license
cc-by-4.0 (IndicCOPA and IndicQA metadata)+cc0-1.0 (Naamapadam metadata; the IndicBERT README says all datasets from the work are CC-0)+not-clearly-stated-on-card (IndicXParaphrase and IndicSentiment: no license in card or API metadata)
access
public (all five repositories ungated)
dataset_card
partial (IndicCOPA card is an empty template and IndicXParaphrase has none; the task list lives in the IndicBERT README)

The test sets download without a gate and the project README dedicates the new sets to the public domain. The Hugging Face repositories disagree with it, two carrying attribution licenses and two carrying none, and most per-task cards are empty.

Adoption

2 high confidence
2.0

Hugging Face downloads summed over the five AI4Bharat task repositories. Naamapadam downloads include use of its training set outside the benchmark, and the third-party tasks are not counted.

Capability

4 medium confidence
4.0

In the typical language IndicXTREME targets, its deepest test has sixteen hundred to sixty-four hundred items that count toward depth. Of its 21 target languages, 19 have a countable test. An item count says how much a set can test, not how precisely it ranks models.

  • https://arxiv.org/abs/2204.08582 recorded 2026-11-08

    "MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages"

  • https://arxiv.org/abs/2212.10168 recorded 2026-11-08

    Naamapadam: training data "automatically created from the Samanantar parallel corpus"; "manually annotated testsets for 9 languages"

  • https://arxiv.org/pdf/1809.05053 recorded 2026-11-18

    "XNLI consists of 7500 human-annotated development and test examples in NLI three-way classification format in English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, Hindi, Swahili and Urdu"; "we create the full XNLI corpus by employing professional translators to translate it into our ten target languages"

  • https://arxiv.org/pdf/2212.05409 recorded 2026-11-18

    Table 2: "|Test| is the size of the test sets in each language. |Dev| is the size of in-language development sets, if available": IndicSentiment 156 dev and 1000 test, IndicXNLI 2490 and 5010 (MT), IndicCOPA 500, IndicXPara 2002, M-Intent and M-SlotFill 2033 and 2974, Naamapadam 607-1080 test, IndicQA 1517-2017, FLORES 1012; Table 7, IndicQA questions per language: as 1789, bn 1763, gu 2017, hi 1547, kn 1517, ml 1589, mr 1604, or 1680, pa 1542, ta 1804, te 1734; Table 9, "Of the 5010 test instances in each language, the number of instances verified and corrected so far is presented in the Ver. and Corr. columns respectively": verified as 3000, bn 1510, hi 4000, kn 1370, ml 3200, mr 1648, or 2107, te 872, and corrected as 1918, bn 835, hi 1142, kn 264, ml 2427, mr 944, or 1820, te 527, none for gu, pa and ta; "We manually translate the COPA (Roemmele et al., 2011) test set into 18 Indic languages"; "All the reviews are first written in English and then manually translated to 13 Indic languages"; "released an automatically translated version of XNLI"; "we manually verify the translations of some parts of the test set and make changes where necessary"; "we machine-translate these sentences into the required languages using the IndicTrans model"; "verify and correct the translations, if required, and (ii) create one paraphrase and a non-paraphrase for each sentence"; "The annotators are asked to create questions (in an Indic language)"; "1012 sentences manually translated into 18 Indic languages"; "we only create test sets"; the per-task results tables (15 to 23) name each task's languages, IndicSentiment's including "bd", which the paper uses for Bodo ("low-resource languages like bd and dg"; Table 14 "brx Bodo"); Table 2 marks IndicXNLI with "ζ - Human verification is in progress, please refer to Appendix I", and Appendix I reads "Our effort to manually correct all the translations in the IndicXNLI (Aggarwal et al., 2022) dataset is currently ongoing" and "Once the complete test set is verified and cleaned, we plan to update IndicXTREME with the additional data"; Table 2 gives the NLI task 12 languages, and Section 3.2 describes IndicXNLI as "an automatically translated version of XNLI (Conneau et al., 2018) in 11 Indic languages"; Table 16 adds "ur"; Table 20, Naamapadam's results, covers "bn gu hi kn ml mr pa ta te"

  • https://arxiv.org/pdf/2212.10168 recorded 2026-11-08

    Table 3 test sentences per language: bn 607, gu 1.1K, hi 867, kn 1.0K, ml 974, mr 1.1K, pa 993, ta 758, te 847; "The Naamapadam-test comprises 500-1000 annotated sentences per language for 9 languages"; "The annotators had to verify if the projected NER annotations were correct and rectify the annotations if incorrect"

  • https://datasets-server.huggingface.co/size?dataset=ai4bharat/naamapadam recorded 2026-11-18

    "test" split sizes: "bn" 607, "gu" 1076, "hi" 867, "kn" 1019, "ml" 974, "mr" 1080, "pa" 993, "ta" 758 and "te" 847, beside "as" 51 and "or" 994

  • https://datasets-server.huggingface.co/size?dataset=Divyanshu/indicxnli recorded 2026-11-18

    split sizes: "test" 5010 rows in each of "as", "bn", "gu", "hi", "kn", "ml", "mr", "or", "pa", "ta" and "te"; "validation" 2490 rows in each except "ta", which has 3238

  • https://huggingface.co/api/datasets/Divyanshu/indicxnli/commits/main recorded 2026-11-18

    the repository's newest commit is "Update README.md" on 2022-10-06, and the data was last added in "add split dataset" on 2022-08-26

  • https://huggingface.co/datasets/ai4bharat/IndicCOPA recorded 2026-11-08

    Collections including ai4bharat/IndicCOPA: "Airavata Evaluation Suite" and "IndicXTREME"

  • https://huggingface.co/datasets/Divyanshu/indicxnli/raw/main/README.md recorded 2026-11-18

    card metadata "annotations_creators:" "machine-generated" and "language_creators:" "machine-generated"; "INDICXNLI include NLI data for eleven major Indic languages", which it lists without Urdu

  • https://raw.githubusercontent.com/AI4Bharat/IndicBERT/main/fine-tuning/xnli/xnli.py recorded 2026-11-18

    IndicXTREME's NLI evaluation script loads Urdu from XNLI and every other language from IndicXNLI: "if args.eval_data == "ur":", "dataset = load_dataset("xtreme", "XNLI")", "dataset = dataset.filter(lambda x: x['language'].startswith('ur'))", and otherwise "dataset = load_dataset("Divyanshu/indicxnli", f"{args.eval_data}", use_auth_token=True)"

  • https://raw.githubusercontent.com/AI4Bharat/IndicBERT/main/README.md recorded 2026-11-18

    "IndicXTREME benchmark includes 9 tasks"; "a human-supervised benchmark, IndicXTREME, consisting of nine diverse NLU tasks covering 20 languages ... 105 evaluation sets, of which 52 are new"; the task list links the NLI set to "https://huggingface.co/datasets/Divyanshu/indicxnli" as "Automatically translated version of XNLI in 11 Indic languages", and Naamapadam as "NER dataset with manually curated testsets for 9 Indic languages"

Verified 2026-09-24, except some scores, still awaiting re-confirmation