Mangosteen
VISTECMangosteen is an open Thai pretraining corpus of about 47 billion tokens. Most of it is Thai Common Crawl and FineWeb2 text passed through a Dolma pipeline adapted for Thai, with rule-based language identification, revised quality filters and filters for gambling and adult content. About 1.5 billion tokens come from curated non-web sources such as Wikipedia, Royal Gazette texts, OCR-extracted books and CC-licensed YouTube subtitles. VISTEC and AI Singapore researchers built it for the WangchanLION v3 model.
AI Singapore publishes the corpus on the Hub as WangchanLION-Web and WangchanLION-Curated. The cleaning pipeline and the full list of curated sources are in the vistec-AI/Mangosteen repository on GitHub.
Openness
5 high confidence- license
- odc-by (card metadata and API tag on both repositories; ODC-By 2.4 leaves rights in the individual pages with their owners. The web share is Thai Common Crawl and FineWeb2 text, and Common Crawl's CCBot checks robots.txt before fetching; the curated share was checked for permissive content licenses)
- access
- public (both repositories ungated on the Hub)
- dataset_card
- present (the web card describes the sources and cleaning pipeline and links the paper; the curated card lists sources by domain with document and word counts)
Both parts download without a gate under ODC-By, and the paper and cards name where the text came from: Thai Common Crawl and FineWeb2 for the web share, with the curated sources listed one by one. As with any web corpus, the license covers the compilation, and each page keeps its own rights.
- https://arxiv.org/pdf/2507.14664 recorded 2026-11-17
"We merged two sources, both Thai Common Crawl and FineWeb2"; "All sources were verified to ensure the use of permissive content licenses. The full list of text sources can be found in https://github.com/vistec-AI/Mangosteen."; "YouTube: We collect Thai YouTube subtitles from videos with the CC BY 3.0 license."
- https://commoncrawl.org/faq recorded 2026-11-17
Common Crawl FAQ: "CCBot is an automated crawler, checking first the robots.txt, and if crawling a page is allowed, fetches pages using HTTP GET requests."
- https://huggingface.co/api/datasets/aisingapore/WangchanLION-Curated recorded 2026-11-17
"gated":false; tag license:odc-by; tag language:th.
- https://huggingface.co/api/datasets/aisingapore/WangchanLION-Web recorded 2026-11-17
"gated":false; tag license:odc-by; tag language:th.
- https://huggingface.co/datasets/aisingapore/WangchanLION-Curated/raw/main/README.md recorded 2026-11-17
"license: odc-by"; train split 397,488 and validation split 4,044 examples; per-source table beginning "encyclopedic | th.wikibooks.org".
- https://huggingface.co/datasets/aisingapore/WangchanLION-Web/raw/main/README.md recorded 2026-11-17
"license: odc-by"; table "CC-Derived Dataset | 29.7 M | 45.9" billion tokens; "We adopt the data collection of Dolma by applying five major components" (language identity, URL deduplication, quality filters, content filters, text-overlap deduplication).
Adoption
2 high confidenceHugging Face downloads summed over the web and curated repositories, most of them for the web share. A pretraining corpus is typically fetched once and reused, so repeat downloads understate how widely a copy is used.
- https://huggingface.co/api/datasets/aisingapore/WangchanLION-Curated recorded 2026-11-17
"downloads":281 for aisingapore/WangchanLION-Curated.
- https://huggingface.co/api/datasets/aisingapore/WangchanLION-Web recorded 2026-11-17
"downloads":868 for aisingapore/WangchanLION-Web.
Capability
3 medium confidenceMangosteen is the largest open Thai pretraining corpus here, several times the Thai share of SEA-PILE, it is documented in a paper that releases its cleaning pipeline, and WangchanLION v3 was trained on it. Its size keeps it at 3: Thai is measured on the same yardstick as English, and forty-seven billion tokens is well short of the hundred billion the next step needs.
- https://arxiv.org/abs/2507.14664 recorded 2026-11-17
Abstract: "We introduce Mangosteen: a 47 billion-token Thai corpus built through a Thai-adapted Dolma pipeline"; "an 8B-parameter SEA-LION model continually pre-trained on Mangosteen then surpasses SEA-LION-v3 and Llama-3.1 by about four points on Thai benchmarks".
- https://huggingface.co/aisingapore/WangchanLION-v3 recorded 2026-11-17
WangchanLION-v3 model card, tagged arxiv:2507.14664 ("Mangosteen: An Open Thai Corpus for Language Model Pretraining"), lists under Resources "Pre-training data (web): https://huggingface.co/datasets/aisingapore/WangchanLION-Web" and "Pre-training data (curated)".
- https://huggingface.co/datasets/aisingapore/SEA-PILE-v2/raw/main/README.md recorded 2026-11-17
Token table row "Thai | th | 6.5" billion tokens.
- https://raw.githubusercontent.com/vistec-AI/Mangosteen/main/README.md recorded 2026-11-17
Table: "Common Crawl-Derived | 29.7M | 45.9", "Curated Non-Common Crawl | 425,304 | 1.5", "Total | 30.1M | 47.4" (documents, billions of tokens).
Verified 2026-11-17, except some scores, still awaiting re-confirmation