The map now includes data for underserved languages.
Gap Map 2 weeks agoWe added a new category to the map, Language-specific datasets, for languages that big AI models tend to underserved. It holds 97 datasets, more than any other category on the map.
AI models need training data in a language, and tests to check how well they work in it. The new category brings both together, including training text in Hindi, Tamil, and Catalan; recordings in Kenyan languages; and tests in Arabic.
Many widely used AI models perform better in English than in underserved languages, partly because they have much more English text to learn from. Sangraha, the largest open collection of training text in the new category, was built by AI4Bharat, a research initiative at IIT Madras in India. It contains about 251 billion tokens, the units of text AI models process, across 22 Indian languages and English. FineWeb, Hugging Face's English-language training collection, contains more than 18.5 trillion tokens, roughly 70 times as much in English alone.
Of the 97 datasets, 28 are fully open, so anyone can download and use them. The other 69 come with conditions, from extra terms you have to check to being closed entirely. Data can exist in a language and still be out of reach for the people who want to build with it.
Explore the new category on the Gap Map to find training data and tests for specific languages and check the conditions for using them.