Back to Gap Map Infrastructure / Core ML frameworks & libraries

Tokenizers

open source / Overall score: 5.0(leading)

Tokenizers is a library providing fast implementations of modern tokenization algorithms including Byte-Pair Encoding, WordPiece, and Unigram. It is written in Rust with bindings for Python and Node.js, and emphasizes high-throughput training and inference. Maintained by Hugging Face, it is a direct dependency of the Transformers library.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core features withheld: no

The repository carries the unmodified Apache License 2.0 and the full Rust and Python source is public. The README describes the whole library with no paid, enterprise or hosted tier beside it, so nothing in the core is withheld.

https://cdn.jsdelivr.net/gh/huggingface/tokenizers@main/LICENSE recorded 2026-09-29
The repository LICENSE file is the Apache License, Version 2.0 text ("Grant of Copyright License. Subject to the terms and conditions of").
https://cdn.jsdelivr.net/gh/huggingface/tokenizers@main/README.md recorded 2026-09-29
The README presents the whole library, "The SOTA tokenization library on all languages, all models, all hardwares.", with the Rust engine and Python and Node.js bindings. Searched the body for paid, enterprise, hosted and cloud: none.
https://github.com/huggingface/tokenizers/blob/main/LICENSE recorded 2026-08-13
LICENSE file is the verbatim Apache License Version 2.0 text
https://pypi.org/pypi/tokenizers/json recorded 2026-09-29
PyPI metadata for tokenizers carries the classifier "License :: OSI Approved :: Apache Software License" and a Source link to the huggingface/tokenizers repository.

Adoption

5 high confidence

5.0

Adoption is measured by PyPI downloads of the tokenizers package.

https://pypistats.org/api/packages/tokenizers/recent recorded 2026-08-13
last_month downloads = 252,452,945 for tokenizers

Capability

5 high confidence

5.0

Tokenizers trains new vocabularies and runs BPE, WordPiece and Unigram tokenization in Rust, with alignment-tracking normalization and truncation, padding and special-token processing. The Transformers library depends on it directly.

https://cdn.jsdelivr.net/gh/huggingface/tokenizers@main/README.md recorded 2026-09-29
The README lists Rust as the "reference implementation" beside Python and Node.js bindings, and describes the tk-train crate as "The `Trainer` trait, every concrete `*Trainer`".
https://github.com/huggingface/tokenizers recorded 2026-08-13
README still lists trainable BPE/WordPiece/Unigram tokenizers with normalization and alignment tracking, in Rust with multi-language bindings
https://pypi.org/pypi/tokenizers/json recorded 2026-09-29
The PyPI page for tokenizers says "Extremely fast (both training and tokenization), thanks to the Rust implementation." and "Normalization comes with alignments tracking."
https://pypi.org/pypi/transformers/json recorded 2026-09-29
The PyPI metadata for transformers lists a hard dependency on the tokenizers package: "tokenizers<0.24.0,>=0.23.1".

Verified 2026-09-24