Back to Gap Map Organization

CIS, LMU Munich

lab · Germany

Scores

1 product on the map — 1 open.

Glot500-c

Openness

4 high confidence
4.0
license
mixed-per-subset (Each sentence keeps its source dataset's license; packaging, metadata and annotations are cc0-1.0. The GitHub LICENSE is Apache-2.0 for code.)
access
public (Ungated for redistributable sources; withheld parts by request form.)
dataset_card
present

The redistributable part downloads without a gate, and every sentence keeps the license of the dataset it came from, which ranges from permissive to non-commercial. Sources that forbid redistribution are withheld and offered through a request form.

Adoption

3 high confidence
3.0

Hugging Face downloads over the trailing month for the Glot500 repository. The withheld parts shared by request are not counted.

Capability

3 medium confidence
3.0

The Glot corpus reaches hundreds of mostly low-resource languages, more than any other open pretraining corpus here, records the source, collection method and license of each part, and trained its builders' Glot encoder. Its size is taken from the public release, because parts of the full corpus cannot be redistributed, and even the full corpus is a sliver beside English pretraining sets such as FineWeb that run to trillions of tokens.

  • https://arxiv.org/abs/2305.12182 recorded 2026-11-12

    "collect and clean Glot500-c, a corpus that covers these 511 languages and allows us to train Glot500-m".

  • https://arxiv.org/pdf/2305.12182 recorded 2026-11-12

    "We train the multilingual model Glot500-m on a 600GB corpus"; Table 1: Glot500-c, 511 languages, 1.5B sentences; "certain sources prohibit the redistribution of data. As such, data from these sources is omitted from the published version of Glot2000-c."

  • https://huggingface.co/datasets/cis-lmu/Glot500 recorded 2026-11-12

    Dataset panel: Number of rows 1,225,300,317; Total file size 308 GB. Card: "Parts of the original Glot500 dataset cannot be published publicly."

  • https://raw.githubusercontent.com/cisnlp/Glot500/main/README.md recorded 2026-11-12

    "Glot500-c is a subset of Glot2000-c for over 500 languages, including languages with more than 30,000 sentences."