Infrastructure / Core ML frameworks & libraries

Tokenizers

Tokenizers is a library providing fast implementations of modern tokenization algorithms including Byte-Pair Encoding, WordPiece, and Unigram. It is written in Rust with bindings for Python, Node.js, and Ruby, and emphasizes high-throughput training and inference. Maintained by Hugging Face, it underpins tokenization in the Transformers library and many production NLP pipelines.

Verified live 2026-06-22 via primary sources. LICENSE body is verbatim Apache License 2.0; full Rust/Python source public.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core-gated: ungated

LICENSE body is verbatim Apache License 2.0; full Rust/Python source public.

https://github.com/huggingface/tokenizers/blob/main/LICENSE recorded 2026-06-22
LICENSE file is the verbatim Apache License Version 2.0

Adoption

5 high confidence

5.0

PyPI last-month downloads of 189,770,639 exceed the 50M+ threshold.

https://pypistats.org/api/packages/tokenizers/recent recorded 2026-06-22
last_month = 189,770,639 PyPI downloads

Capability

5 high confidence

5.0

Broad feature set and central dependency of the Transformers ecosystem.

https://github.com/huggingface/tokenizers recorded 2026-06-22
README lists BPE/WordPiece/Unigram, normalization, multi-language bindings

Unchanged since 2026-07-30 (last edited, not re-checked)