bytedance/1d-tokenizer
This repo contains a 1D tokenizer and generator, potentially useful for AI coding tools and research.
Jupyter Notebook
AI & Machine Learning
LLM Frameworks
Apache-2.0
1.1K
Stars
65
Forks
Jun 21, 2024
Created
Mar 20, 2025
Last Updated
Project Analytics
Stars Growth (1 Month)
+10
+0.9% change
Avg Daily Growth (1 Month)
+0.4
stars per day
Fork/Star Ratio (All Time)
5.8%
Normal engagement
Lifetime Growth
1.8
stars/day over 624 days
Stars Over Time
Forks Over Time
Open Issues Over Time
Pull Requests Over Time
Commits Over Time
Topics
AI-Generated Tags
tokenization
text-generation
ai-research
command-line-interface
python-library
Similar Projects
EbookFoundation/free-programming-books
Free programming books repository with AI-powered features
383.6K
Python
jwasham/coding-interview-university
A comprehensive computer science study plan for software engineers, with a focus on coding interviews and AI-related topics.
337.6K
vinta/awesome-python
Awesome Python framework and resource collection
285.7K
Python
openclaw/openclaw
Personal AI assistant for messaging platforms, self-hosted and multi-channel
265.0K
TypeScript
Comments (0)
Sign in to leave a comment or vote
Sign InNo comments yet. Be the first to comment!
Stay in the loop
Get weekly updates on trending AI coding tools and projects.