Explore Projects
Discover 170 open source projects
Showing 61-80 of 170 projects
Boris-code/feapder
A powerful Python-based web scraping framework with features like browser rendering and data deduplication.
ScottSloan/Bili23-Downloader
A cross-platform Bilibili video downloader with multi-thread acceleration and resumable downloads.
elliotgao2/toapi
A Python library that makes it easy to create APIs for any website, simplifying web scraping and API development.
Gerapy/Gerapy
A distributed crawler management framework based on Scrapy, Scrapyd, Django, and Vue.js for web scraping.
google/robotstxt
A C++ library for parsing and matching Google's robots.txt files, useful for web crawlers and bots.
ngc660sec/NGCBot
A feature-rich WeChat bot with AI-powered capabilities for developers, including security news, vulnerability lookup, and more.
jasonxtn/Argus
A comprehensive toolkit for information gathering and reconnaissance, including OSINT, web crawling, and more.
internetarchive/heritrix3
Heritrix is an open-source, extensible web crawler for archiving websites at scale.
apache/nutch
Apache Nutch is an extensible and scalable web crawler for building search engines and data mining applications.
CrawlScript/WebCollector
An open-source web crawler framework written in Java that makes it easy to build multi-threaded web crawlers.
jaeles-project/gospider
Fast web spider written in Go for bug bounty
JAVClub/core
This repository is a web crawler for adult video streaming and torrent sites, not a developer tool.
CharlesPikachu/DecryptLogin
Comprehensive Python library for logging in to various websites using the requests library.
NikolaiT/GoogleScraper
A Python module to scrape several search engines, including asynchronous networking support.
brianway/webporter
A Java web crawler application built using the webmagic framework, focused on indexing content from Zhihu.
geziyor/geziyor
Geziyor is a fast web crawling and scraping framework for Go that supports JavaScript rendering.
facundoolano/google-play-scraper
A Node.js library for scraping data from the Google Play Store.
any4ai/AnyCrawl
AnyCrawl is a Node.js/TypeScript web scraper that extracts structured data from search engines and websites for use in AI/LLM applications.
jae-jae/QueryList
A progressive PHP crawler framework that allows developers to build elegant web scrapers and crawlers.
loadchange/amemv-crawler
A Python library for easily downloading videos from TikTok (Douyin), useful for content creators and researchers.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.