Language datasets for underrepresented languages are now on the map

Language-Specific Datasets

We added language-specific datasets as a new category, with 97 products covering text, speech, instruction and evaluation data for underrepresented languages. You’ll see it open at stage 4 with a resiliency gap. The largest open corpus we found for an underrepresented language is Sangraha, at about a quarter of a trillion tokens across 22 Indian languages. FineWeb’s English corpus alone is 18.5 trillion.

The map now covers 862 products across 25 categories.

Fixes & Minor Updates

• Data licences: We now score data licences on the same 1 to 5 scale we use for models and software. Published data with no stated licence scores 2, and we keep 1 for data that is actually closed. 14 dataset scores moved from 3 to 2.
• CC-BY weights: We now count model weights released under CC-BY as permissive. We’ve also moved FSL into the competition-restricted tier.
• lakeFS: We re-scored lakeFS after it changed its licence to BSL 1.1.