huggingface/tokenizers
Fast, state-of-the-art tokenizers optimized for research and production in AI/ML applications
Rust
AI & Machine Learning
LLM Frameworks
Apache-2.0
10.5K
Stars
1.0K
Forks
Nov 1, 2019
Created
Feb 28, 2026
Last Updated
Project Analytics
Stars Growth (1 Month)
+53
+0.5% change
Avg Daily Growth (1 Month)
+1.9
stars per day
Fork/Star Ratio (All Time)
10.0%
Normal engagement
Lifetime Growth
4.5
stars/day over 2.3K days
Stars Over Time
Forks Over Time
Open Issues Over Time
Pull Requests Over Time
Commits Over Time
AI-Generated Tags
tokenizers
natural-language-processing
transformers
bert
gpt
Similar Projects
EbookFoundation/free-programming-books
Free programming books repository with AI-powered features
383.6K
Python
jwasham/coding-interview-university
A comprehensive computer science study plan for software engineers, with a focus on coding interviews and AI-related topics.
337.6K
vinta/awesome-python
Awesome Python framework and resource collection
285.7K
Python
openclaw/openclaw
Personal AI assistant for messaging platforms, self-hosted and multi-channel
265.0K
TypeScript
Comments (0)
Sign in to leave a comment or vote
Sign InNo comments yet. Be the first to comment!
Stay in the loop
Get weekly updates on trending AI coding tools and projects.