Explore Projects
Discover 170 open source projects
Showing 41-60 of 170 projects
rmax/scrapy-redis
A Redis-based distributed scraping library for the Scrapy web crawling framework.
SpiderClub/haipproxy
A high-availability distributed IP proxy pool powered by Scrapy and Redis for web crawling applications.
adbar/trafilatura
Gathers text and metadata from the web using crawling, scraping, and extraction techniques.
DropsDevopsOrg/ECommerceCrawlers
A collection of Python-based web crawlers for scraping data from various e-commerce and online platforms.
hect0x7/JMComic-Crawler-Python
Python API for crawling and downloading content from the JMComic website, a popular source for adult manga/comics.
hakluke/hakrawler
A fast, simple web crawler designed for quick discovery of endpoints and assets within a web application.
Alfred1984/interesting-python
This GitHub repository contains a collection of interesting Python web scraping and data analysis projects.
niespodd/browser-fingerprinting
Analysis of bot protection systems and techniques to bypass browser fingerprinting for web scraping.
SpiderClub/weibospider
A distributed web crawler for Weibo, built using Celery and Requests.
HiddenStrawberry/Crawler_Illegal_Cases_In_China
A repository that collects news, resources, and legal regulations related to web crawlers in China.
myreader-io/myGPTReader
A community-driven platform to read and chat with AI bots powered by ChatGPT for developers.
hanc00l/wooyun_public
This is an archived repository that provides a web crawler and search engine for the now-defunct Wooyun security vulnerability database.
dataabc/weibo-crawler
A Python-based web crawler that can scrape data, images, and videos from Weibo, a popular social media platform in China.
Arachni/arachni
Arachni is a powerful open-source web application security scanner framework for penetration testing and vulnerability detection.
kanasimi/work_crawler
A JavaScript crawler that downloads comics, novels, and webcomics from various online sources.
bitmagnet-io/bitmagnet
Self-hosted BitTorrent indexer, crawler, classifier and search engine with web UI and API
hardkoded/puppeteer-sharp
Headless Chrome .NET API for web automation, crawling, and end-to-end testing
DedSecInside/TorBot
DedSecInside/TorBot is a dark web OSINT tool written in Python that crawls and extracts information from the Tor network.
NanmiCoder/CrawlerTutorial
A Python crawler tutorial for beginners, intermediate, and advanced users.
ai-robots-txt/ai.robots.txt
A list of AI agents and robots to block, useful for privacy-conscious developers
Stay in the loop
Get weekly updates on trending AI coding tools and projects.