Tokenizers
Hugging FaceTokenizers is a library providing fast implementations of modern tokenization algorithms including Byte-Pair Encoding, WordPiece, and Unigram. It is written in Rust with bindings for Python, Node.js, and Ruby, and emphasizes high-throughput training and inference. Maintained by Hugging Face, it underpins tokenization in the Transformers library and many production NLP pipelines.
Verified 2026-08-13 via GitHub and the LICENSE body.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public
- core-gated
- ungated
LICENSE body is verbatim Apache License 2.0; full Rust/Python source public. The repository is public and unarchived, and the README describes the whole library with no paid, enterprise or hosted tier beside it, so source is public and the core ungated.
- https://github.com/huggingface/tokenizers/blob/main/LICENSE recorded 2026-08-13
LICENSE file is the verbatim Apache License Version 2.0 text
- https://api.github.com/repos/huggingface/tokenizers recorded 2026-08-13
Repo metadata - license spdx_id Apache-2.0, private false, archived false, default branch main - for huggingface/tokenizers.
- https://raw.githubusercontent.com/huggingface/tokenizers/main/README.md recorded 2026-08-13
README lists the trainable BPE/WordPiece/Unigram implementations and the Rust/Python/Node/Ruby bindings as published. No paid tier appears.
Adoption
5 high confidence252,452,945 PyPI downloads of `tokenizers` in the trailing 30 days, which falls in the top band of the software usage scale, over 10M a month, level 5.
- https://pypistats.org/api/packages/tokenizers/recent recorded 2026-08-13
last_month downloads = 252,452,945 for tokenizers
Capability
5 high confidenceBroad feature set and central dependency of the Transformers ecosystem.
- https://github.com/huggingface/tokenizers recorded 2026-08-13
README still lists trainable BPE/WordPiece/Unigram tokenizers with normalization and alignment tracking, in Rust with multi-language bindings
Verified 2026-08-13