Explore Projects
Discover 96 open source projects
Showing 21-40 of 96 projects
MontFerret/ferret
Declarative web scraping library written in Go, providing a powerful DSL for extracting data from websites.
yujiosaka/headless-chrome-crawler
A powerful, distributed web crawler powered by Headless Chrome for scraping websites at scale.
rmax/scrapy-redis
A Redis-based distributed scraping library for the Scrapy web crawling framework.
SpiderClub/haipproxy
A high-availability distributed IP proxy pool powered by Scrapy and Redis for web crawling applications.
adbar/trafilatura
Gathers text and metadata from the web using crawling, scraping, and extraction techniques.
hect0x7/JMComic-Crawler-Python
Python API for crawling and downloading content from the JMComic website, a popular source for adult manga/comics.
hakluke/hakrawler
A fast, simple web crawler designed for quick discovery of endpoints and assets within a web application.
lc/gau
A Go-based tool to fetch known URLs from various threat intelligence sources for security analysis.
201206030/novel-plus
This is a comprehensive content management system for novels, including features like recommendation, search, reading, and more.
hanc00l/wooyun_public
This is an archived repository that provides a web crawler and search engine for the now-defunct Wooyun security vulnerability database.
omkarcloud/botasaurus
Powerful scraping framework to build undetectable web scrapers using Python
exa-labs/exa-mcp-server
A TypeScript-based server for web search and web crawling, part of the Exa MCP ecosystem.
hardkoded/puppeteer-sharp
Headless Chrome .NET API for web automation, crawling, and end-to-end testing
DedSecInside/TorBot
DedSecInside/TorBot is a dark web OSINT tool written in Python that crawls and extracts information from the Tor network.
ai-robots-txt/ai.robots.txt
A list of AI agents and robots to block, useful for privacy-conscious developers
jasonxtn/Argus
A comprehensive toolkit for information gathering and reconnaissance, including OSINT, web crawling, and more.
apache/nutch
Apache Nutch is an extensible and scalable web crawler for building search engines and data mining applications.
CrawlScript/WebCollector
An open-source web crawler framework written in Java that makes it easy to build multi-threaded web crawlers.
crawl/crawl
This is the official repository for the classic roguelike game Dungeon Crawl: Stone Soup, not a vibe coder platform.
geziyor/geziyor
Geziyor is a fast web crawling and scraping framework for Go that supports JavaScript rendering.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.