SentencePiece

license

Apache-2.0(OSI)

source

public

core-gated

ungated

LICENSE body is verbatim Apache License 2.0; full C++/Python source public. The repository is public and unarchived, and the README describes the whole library with no paid, enterprise or hosted tier beside it, so source is public and the core ungated.

• https://github.com/google/sentencepiece/blob/master/LICENSE recorded 2026-08-13

  LICENSE file is the verbatim Apache License Version 2.0 text

• https://api.github.com/repos/google/sentencepiece recorded 2026-08-13

  Repo metadata - license spdx_id Apache-2.0, private false, archived false, default branch master - for google/sentencepiece.

• https://raw.githubusercontent.com/google/sentencepiece/master/README.md recorded 2026-08-13

  README describes the unsupervised subword tokenizer/detokenizer, with the C++ build and the Python wheel both published. No paid tier appears.

36,642,180 PyPI downloads of `sentencepiece` in the trailing 30 days, which clears the top band of the software usage scale, over 10M a month, level 5. That is the same band other products on this map hold at comparable volume - pydantic-ai at around 31M, langgraph at around 58M. SentencePiece is the de facto subword tokenizer underneath most LLM and machine-translation pipelines. The C++ library is also vendored directly into other projects and goes uncounted here, so the figure is a floor within the top band.

• https://pepy.tech/projects/sentencepiece recorded 2026-08-13

  36,642,180 downloads in the last 30 days for sentencepiece

Widely used tokenizer underpinning many LLM and NMT models, though narrow in scope.

• https://github.com/google/sentencepiece recorded 2026-08-13

  README still describes language-independent subword tokenization (BPE and unigram) trained from raw text