- Repository
- https://github.com/huggingface/tokenizers
- Homepage
- https://huggingface.co/docs/tokenizers
- Owner
- huggingface (Organization)
- Stars
- 11,183
- Forks
- 1,226
- Open issues
- 198
- Language
- Rust
- License
- Apache-2.0
- Created
- 2019-11-01
- Last push
- 2026-10-18
- Topics
- bert, gpt, language-model, natural-language-processing, natural-language-understanding, nlp, transformers
- Snapshot
- 2026-10-18
GitHub · #716
huggingface/tokenizers
huggingface/tokenizers is #716 in this GitHub directory, with 11,183 stars. 💥 Fast State-of-the-Art Tokenizers optimized for Research and Production