Back to Gap Map Model components / Language-specific datasets

Mangosteen

VISTEC
open / Overall score: 2.7

Mangosteen is an open Thai pretraining corpus of about 47 billion tokens. Most of it is Thai Common Crawl and FineWeb2 text passed through a Dolma pipeline adapted for Thai, with rule-based language identification, revised quality filters and filters for gambling and adult content. About 1.5 billion tokens come from curated non-web sources such as Wikipedia, Royal Gazette texts, OCR-extracted books and CC-licensed YouTube subtitles. VISTEC and AI Singapore researchers built it for the WangchanLION v3 model.

AI Singapore publishes the corpus on the Hub as WangchanLION-Web and WangchanLION-Curated. The cleaning pipeline and the full list of curated sources are in the vistec-AI/Mangosteen repository on GitHub.

Openness

5 high confidence
5.0
license
odc-by (card metadata and API tag on both repositories; ODC-By 2.4 leaves rights in the individual pages with their owners. The web share is Thai Common Crawl and FineWeb2 text, and Common Crawl's CCBot checks robots.txt before fetching; the curated share was checked for permissive content licenses)
access
public (both repositories ungated on the Hub)
dataset_card
present (the web card describes the sources and cleaning pipeline and links the paper; the curated card lists sources by domain with document and word counts)

Both parts download without a gate under ODC-By, and the paper and cards name where the text came from: Thai Common Crawl and FineWeb2 for the web share, with the curated sources listed one by one. As with any web corpus, the license covers the compilation, and each page keeps its own rights.

Adoption

2 high confidence
2.0

Hugging Face downloads summed over the web and curated repositories, most of them for the web share. A pretraining corpus is typically fetched once and reused, so repeat downloads understate how widely a copy is used.

Capability

3 medium confidence
3.0

Mangosteen is the largest open Thai pretraining corpus here, several times the Thai share of SEA-PILE, it is documented in a paper that releases its cleaning pipeline, and WangchanLION v3 was trained on it. Its size keeps it at 3: Thai is measured on the same yardstick as English, and forty-seven billion tokens is well short of the hundred billion the next step needs.

Verified 2026-11-17, except some scores, still awaiting re-confirmation