google/robotstxt

A C++ library for parsing and matching Google's robots.txt files, useful for web crawlers and bots.

C++
Backend & APIs
API Frameworks
Apache-2.0

3.5K

Stars

247

Forks

Jun 11, 2019

Created

Feb 27, 2026

Last Updated

Project Analytics

Stars Growth (1 Month)

+7

+0.2% change

Avg Daily Growth (1 Month)

+0.3

stars per day

Fork/Star Ratio (All Time)

7.1%

Normal engagement

Lifetime Growth

1.4

stars/day over 2.5K days

Stars Over Time

Forks Over Time

Open Issues Over Time

Pull Requests Over Time

Commits Over Time

AI-Generated Tags

robots.txt
parsing
matching
web-crawler
bot

Comments (0)

Sign in to leave a comment or vote

Sign In

No comments yet. Be the first to comment!

Stay in the loop

Get weekly updates on trending AI coding tools and projects.