Back to Gap Map Model components / Language-specific evaluation sets

INCLUDE

Cohere
open / Overall score: 3.0

INCLUDE is a multilingual knowledge and reasoning benchmark of four-option multiple-choice questions taken from local academic, professional and licensing exams in 44 languages. Because the questions come from each region's own exams rather than translated English tests, many probe regional knowledge. A smaller lite subset covers the same languages. It was built by EPFL and Cohere Labs researchers.

Openness

5 high confidence
5.0
license
apache-2.0
access
public
dataset_card
present

Both repositories download from Hugging Face without a gate under Apache-2.0, and the full question set including answers is published.

Adoption

3 high confidence
3.0

Hugging Face downloads over the trailing month for the base and lite repositories combined. Runs through evaluation harnesses that cache the data are counted only when they download it.

Capability

3 medium confidence
3.0

In the typical language INCLUDE targets, its deepest test has four hundred to sixteen hundred items that count toward depth. In 2 of its 44 languages a test taken from another dataset counts for less, because nothing independent backs it there. An item count says how much a set can test, not how precisely it ranks models.

  • https://arxiv.org/abs/2411.19799 recorded 2026-11-08

    "we construct an evaluation suite of 197,243 QA pairs from local exam sources".

  • https://arxiv.org/html/2411.19799 recorded 2026-11-08

    "All questions in the dataset are presented in their native languages and scripts."; regional resources "originally created by native speakers of each country’s official languages. This collection avoids translationese"; Include-base "uniformly samples 22,635 QA pairs"; "Each language has a maximum of 550 samples"; Include-lite is "uniformly drawn from" Include-base "with a total of 10,770 samples"; Section 3.1: "we also consolidate existing datasets with extensive domain coverage in single non-English languages", Table 15 naming ArabicMMLU for Arabic and TurkishMMLU for Turkish; Table 14 and Figure 9 cover the "languages where both existing benchmark data and newly collected data are available"

  • https://datasets-server.huggingface.co/size?dataset=CohereLabs/include-base-44 recorded 2026-11-08

    the Hub's split sizes per language config, test: Albanian 551, Arabic 552, Armenian 550, Azerbaijani 548, Basque 500, Belarusian 550, Bengali 548, Bulgarian 550, Chinese 545, Croatian 550, Dutch 551, Estonian 224, Finnish 551, French 419, Georgian 500, German 139, Greek 552, Hebrew 550, Hindi 547, Hungarian 550, Indonesian 550, Italian 548, Japanese 501, Kazakh 500, Korean 500, Lithuanian 534, Malay 501, Malayalam 479, Nepali 500, North Macedonian 551, Persian 548, Polish 548, Portuguese 551, Russian 552, Serbian 550, Spanish 550, Tagalog 500, Tamil 550, Telugu 548, Turkish 548, Ukrainian 550, Urdu 352, Uzbek 551, Vietnamese 550 (22,639 in all); validation: Albanian 25, Arabic 30, Armenian 20, Azerbaijani 30, Basque 5, Belarusian 15, Bengali 20, Bulgarian 15, Chinese 40, Croatian 15, Dutch 25, Estonian 25, Finnish 25, French 25, Georgian 5, German 15, Greek 35, Hebrew 10, Hindi 40, Hungarian 15, Indonesian 25, Italian 30, Japanese 15, Kazakh 5, Korean 10, Lithuanian 25, Malay 15, Malayalam 30, Nepali 10, North Macedonian 20, Persian 25, Polish 15, Portuguese 30, Russian 40, Serbian 15, Spanish 20, Tagalog 10, Tamil 10, Telugu 20, Turkish 20, Ukrainian 15, Urdu 15, Uzbek 20, Vietnamese 15; the "Dutch - Flemish" and "Dutch-Flemish" configs are listed as failed

  • https://datasets-server.huggingface.co/size?dataset=CohereLabs/include-lite-44 recorded 2026-11-08

    the Hub's split sizes per language config, test only: Albanian 251, Arabic 250, Armenian 251, Azerbaijani 250, Basque 250, Belarusian 250, Bengali 249, Bulgarian 250, Chinese 245, Croatian 250, Dutch 251, Estonian 183, Finnish 251, French 250, Georgian 250, German 89, Greek 253, Hebrew 250, Hindi 245, Hungarian 250, Indonesian 248, Italian 251, Japanese 249, Kazakh 250, Korean 250, Lithuanian 250, Malay 249, Malayalam 252, Nepali 250, North Macedonian 251, Persian 248, Polish 250, Portuguese 253, Russian 252, Serbian 250, Spanish 250, Tagalog 250, Tamil 250, Telugu 249, Turkish 249, Ukrainian 250, Urdu 250, Uzbek 251, Vietnamese 250 (10,770 in all)

  • https://huggingface.co/datasets/CohereLabs/include-base-44/raw/main/README.md recorded 2026-11-08

    Model performance table: Llama3.1-70B-Instruct 70.6, Qwen2.5-14B 62.3, Aya-expanse-32b 59.1.

  • https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/include/default/Albanian/_albanian_template_yaml recorded 2026-11-08

    the harness's per-language INCLUDE template: "dataset_path: CohereForAI/include-base-44", "test_split: test", with no few-shot split in the zero-shot default task

  • https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/include/README.md recorded 2026-11-08

    "INCLUDE is a comprehensive knowledge- and reasoning-centric benchmark across 44 languages"; points to CohereForAI/include-base-44.

Verified 2026-09-24