Developed and maintained by Google's language research team.

SentencePiece is an unsupervised, language independent subword tokenizer and detokenizer trained directly from raw text.

Hosted on GitHub at google/sentencepiece and distributed via PyPI, pip and C++ bindings.

Released in 2018 and now in a stable, low churn maintenance phase.

It lets teams train a consistent subword vocabulary directly from raw text without language specific pre-tokenization rules, unlike earlier tokenizers.

Implements BPE and unigram language model algorithms and is used by T5, ALBERT, Llama and many other models.

Apache 2.0 licensed, open source, maintained by Google, roughly 11k GitHub stars, stable and mostly in maintenance mode with low commit velocity.