Back to Gap Map Model components / Training & synthetic datasets

MADLAD-400

open / Overall score: 3.5

Document-level multilingual web corpus derived from Common Crawl and covering 419 languages, built by Google researchers and hosted by the Allen Institute for AI. It ships in a noisy variant filtered only by language identification and a clean quality-filtered variant of roughly 3 trillion tokens, with a focus on low-resource languages. It underpins the MADLAD-400 machine-translation models.

Openness

5 high confidence

5.0

license: odc-by(API tag
gated: false
card: present

Ungated with a permissive open-data license and a complete dataset card.

https://huggingface.co/api/datasets/allenai/MADLAD-400 recorded 2026-09-29
The Hub API record for allenai/MADLAD-400 lists tags "task_categories:text-generation" and "license:odc-by" and reports the dataset as public and not gated.
https://huggingface.co/datasets/allenai/MADLAD-400/raw/main/README.md recorded 2026-09-29
The raw card front matter reads "license: odc-by" and its License section says "This data is released with the" CC-BY-4.0 license, alongside sections on filtering and a language audit.

Adoption

3 high confidence

3.0

Adoption is measured by Hugging Face downloads of the dataset over the trailing month, the only usage channel on record for MADLAD-400.

https://huggingface.co/api/datasets/allenai/MADLAD-400 recorded 2026-08-13
15,089 downloads in the trailing 30 days for allenai/MADLAD-400

Capability

4 high confidence

4.0

Google trained 3B, 7.2B and 10.7B translation models on this corpus plus other public data, and the 10.7B model, trained on 250 billion tokens covering over 450 languages, is reported as competitive with much larger models. That is a strong documented downstream result rather than a current lead.

https://arxiv.org/abs/2309.04662 recorded 2026-09-29
The arXiv abstract says the authors "train and release a 10.7B-parameter multilingual machine translation model on 250 billion tokens" and find it "competitive with models that are significantly larger".
https://huggingface.co/google/madlad400-10b-mt/raw/main/README.md recorded 2026-09-29
The Google MADLAD-400-10B-MT model card says the model was "trained on 250 billion tokens covering over 450 languages" and "It is competitive with models that are significantly larger."

Verified 2026-09-27