Library / SDK
Inference & ServingFine-tuning & Training
Tokenizers
1Fast 2Rust 3based 4tokenizers 5for 6transformers
the six Ws · specification
W1
Who
Hugging Face maintains Tokenizers as an open source Rust library with Python bindings.
W2
What
Tokenizers implements fast, production ready tokenization algorithms including BPE, WordPiece and Unigram.
W3
Where
Hosted on GitHub at huggingface/tokenizers and distributed via PyPI and crates.io.
W4
When
Released in 2019 and actively maintained.
W5
Why
It removes the tokenization bottleneck in NLP pipelines by processing text far faster than pure Python implementations.
W6
With
Written in Rust with Python bindings and used internally by Transformers.
W7
Watch
Apache 2.0 licensed, open source, maintained by Hugging Face, roughly 11k GitHub stars.
alternatives
works with