danielmiessler/RobotsDisallowed
A curated list of commonly disallowed robots.txt directories, useful for web crawlers and SEO.
Shell
Developer Tools
CLI Tools
1.5K
Stars
304
Forks
Sep 21, 2015
Created
Aug 22, 2022
Last Updated
Project Analytics
Stars Growth (1 Month)
-1
-0.1% change
Avg Daily Growth (1 Month)
-0.0
stars per day
Fork/Star Ratio (All Time)
20.5%
High engagement
Lifetime Growth
0.4
stars/day over 3.8K days
Stars Over Time
Forks Over Time
Open Issues Over Time
Pull Requests Over Time
Commits Over Time
AI-Generated Tags
web-crawling
robots.txt
seo
web-scraping
cli-tool
Similar Projects
ohmyzsh/ohmyzsh
Enhances shell experience with plugins/themes for zsh
185.1K
Shell
microsoft/vscode
Visual Studio Code - Open Source code editor
182.3K
TypeScript
github/gitignore
Official .gitignore templates for GitHub repositories
172.8K
jlevy/the-art-of-command-line
Comprehensive command line guide for developers
160.1K
Comments (0)
Sign in to leave a comment or vote
Sign InNo comments yet. Be the first to comment!
Stay in the loop
Get weekly updates on trending AI coding tools and projects.