Back to Gap Map Model components / Language-specific evaluation sets

IndicGenBench

Google Research
restricted / Overall score: 2.7

IndicGenBench is a benchmark of generation tasks for large language models in 29 Indic languages across 13 scripts, covering cross-lingual summarization, machine translation, and multilingual and cross-lingual question answering. It extends CrossSum, FLORES, XQuAD and XorQA by having people translate their English examples into each target language, which makes the test data multi-way parallel. Google Research releases it as four task sets.

Openness

2 high confidence
2.0
license
cc-by-sa-4.0 (Flores-IN and XQuAD-IN)+mit (XorQA-IN)+cc-by-nc-sa-4.0 (CrossSum-IN)
access
public (all four Hugging Face repositories gated: false; also in the GitHub repository)
dataset_card
present (GitHub README and per-task cards describe tasks, fields and splits)

Each task set keeps the license of the dataset it extends, so the summarization set is non-commercial while the translation and question-answering sets allow commercial use. All four download without a gate.

Adoption

2 high confidence
2.0

Hugging Face downloads summed over the four task repositories. The same data is also cloned from GitHub, which publishes no count.

Capability

3 medium confidence
3.0

In the typical language IndicGenBench targets, its deepest test has four hundred to sixteen hundred items that count toward depth. In one of its 29 languages a test taken from another dataset counts for less, because nothing independent backs it there. An item count says how much a set can test, not how precisely it ranks models.

  • https://arxiv.org/abs/2404.16816 recorded 2026-11-08

    Abstract: "the largest benchmark for evaluating LLMs on user-facing generation tasks across a diverse set 29 of Indic languages covering 13 scripts".

  • https://arxiv.org/pdf/2404.16816 recorded 2026-11-08

    CrossSum-IN: "We sample 700 English article-summary pairs (100 each from train/dev and 500 from test) and ask human translators to translate the English summary into the target Indic languages"; "CrossSum already contains data for 9 of our 29 target languages; for these languages we sample 100/100/500 examples from the original dataset". FLORES-IN: "where the same source English sentences are translated by humans into the target 200 languages"; "It contains data in 22 of our 29 target languages; we extend this by collecting human translations for the remaining 7 new languages"; "We collect translations for all 997 dev and 1012 devtest sentences, yielding 2009 sentences per language". XQuAD-IN: "we use the 240 passages and 1190 QA pairs from XQuAD as our test set"; "We additionally selected 20 passages and 100 QA pairs from the original SQuAD v1.1 training and development sets each to create our training and development set"; "For all the 280 passages and 1390 QA pairs we collect professional human translations in 12 Indic languages". XorQA-IN: "we select the 302 Bengali and 237 Telugu examples"; "dev set as our test data"; "to create our training (100) and development (500) set"; "We collect data for 28 Indic languages"; "We do not collect translations for Nepali." Table 1 gives 14.5k, 29.3k, 14.3k and 15.1k test instances; Table 9 gives each task's languages

  • https://raw.githubusercontent.com/csebuetnlp/CrossSum/main/README.md recorded 2026-11-08

    the language table's header row, "Language (ISO 639-1-Code) | am | ar | az | bn | my | zh-CN | zh-TW | en | fr | gu | ha | hi | ig | id | ja | rn | ko | ky | mr | np | om | ps | fa | pcm | pt | pa | ru | gd | sr | sr | si | so | es | sw | ta | te | th | ti | tr | uk | ur | uz | vi | cy | yo": of IndicGenBench's 29 languages, CrossSum has bn, gu, hi, mr, np, ps, pa, ta, te and ur

  • https://raw.githubusercontent.com/google-research-datasets/indic-gen-bench/main/README.md recorded 2026-11-08

    "the 'dev' and 'test' split of flores_in corresponds to the 'dev' and 'devtest' split of the original flores dataset"; "`split`: one of `{train, dev, test}`"

Verified 2026-09-24