Infrastructure / Core ML frameworks & libraries

SentencePiece

SentencePiece is an unsupervised text tokenizer and detokenizer that implements subword units (Byte-Pair Encoding and unigram language model) directly from raw text, without language-specific pre-tokenization. It treats input as a stream of Unicode characters, enabling a language-independent, end-to-end pipeline useful for languages without clear word boundaries. It is developed and maintained by Google.

Verified live 2026-06-22 via primary sources. LICENSE body is verbatim Apache License 2.0; full C++/Python source public.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core-gated: ungated

LICENSE body is verbatim Apache License 2.0; full C++/Python source public.

https://github.com/google/sentencepiece/blob/master/LICENSE recorded 2026-06-22
LICENSE file is the verbatim Apache License Version 2.0

Adoption

5 high confidence

5.0

Re-scored 4->5. 37.65M PyPI downloads in the last 30 days. The map's prevailing usage_volume level-5 floor is >10M/mo (cf. pydantic-ai ~31M, langgraph ~58M at level 5); the prior note bucketed it into a 5M-50M=L4 band used only in one ml_frameworks batch, leaving a 37M/mo tokenizer below map peers of equal or lower volume. SentencePiece is the de facto subword tokenizer underpinning most LLM/NMT pipelines and clears the prevailing floor.

https://pepy.tech/projects/sentencepiece recorded 2026-06-25
37,654,106 downloads in last 30 days

Capability

4 high confidence

4.0

Widely used tokenizer underpinning many LLM and NMT models, though narrow in scope.

https://github.com/google/sentencepiece recorded 2026-06-22
README describes BPE and unigram subword units, language-independent tokenization

Unchanged since 2026-07-30 (last edited, not re-checked)