SentencePiece
GoogleSentencePiece is an unsupervised text tokenizer and detokenizer that implements subword units (Byte-Pair Encoding and unigram language model) directly from raw text, without language-specific pre-tokenization. It treats input as a stream of Unicode characters, enabling a language-independent, end-to-end pipeline useful for languages without clear word boundaries. The project is hosted under Google's GitHub organization and describes itself as not an official Google product.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public
- core features withheld
- no
The LICENSE file is the full Apache License 2.0 text, and the C++ and Python source is public. The README describes the whole library with no paid, enterprise or hosted tier beside it, and no model weights or license keys are needed, so nothing in the core is gated.
- https://cdn.jsdelivr.net/gh/google/sentencepiece@master/LICENSE recorded 2026-09-29
The file is the full Apache License, "Version 2.0, January 2004" text, including the appendix "APPENDIX: How to apply the Apache License to your work." with no added terms.
- https://cdn.jsdelivr.net/gh/google/sentencepiece@master/README.md recorded 2026-09-29
The README states "SentencePiece is licensed under the [Apache 2.0 License](LICENSE)." and links a "Command Line Interface (CLI) & Build Guide (CMake)" for building the public source; searched the body for paid, enterprise, hosted, license key and subscription: none.
- https://github.com/google/sentencepiece/blob/master/LICENSE recorded 2026-08-13
LICENSE file is the verbatim Apache License Version 2.0 text
- https://pypi.org/pypi/sentencepiece/json recorded 2026-09-29
The PyPI metadata lists "license_expression": "Apache-2.0" for sentencepiece and a project URL to the public repository "https://github.com/google/sentencepiece".
Adoption
5 high confidenceAdoption is measured by PyPI downloads of the sentencepiece package, though the C++ library is also vendored directly into other projects, a use the count does not capture. SentencePiece is the de facto subword tokenizer underneath most LLM and machine-translation pipelines, so the measured volume is a floor rather than a full count.
- https://pepy.tech/projects/sentencepiece recorded 2026-08-13
36,642,180 downloads in the last 30 days for sentencepiece
Capability
4 high confidenceWidely used tokenizer underpinning many LLM and NMT models, though narrow in scope.
- https://cdn.jsdelivr.net/gh/google/sentencepiece@master/README.md recorded 2026-09-29
The README describes BPE and unigram tokenizers trained "directly from raw sentences" giving a "purely end-to-end, language-independent pipeline", subword sampling (Subword Regularization, BPE-Dropout), and a zero-dependency C++20 Lite runtime.
- https://github.com/google/sentencepiece recorded 2026-08-13
README still describes language-independent subword tokenization (BPE and unigram) trained from raw text
Verified 2026-09-24