Explore Projects
Discover 96 open source projects
Showing 61-80 of 96 projects
Python3Spiders/WeiboSuperSpider
A powerful Python-based web crawler and toolkit for scraping Weibo data, including user profiles, comments, images, and more.
chriskite/anemone
Anemone is a Ruby web-spider framework for crawling and extracting data from websites.
rushter/selectolax
A fast HTML5 parser with CSS selectors for Python, useful for web scraping and crawling tasks.
ArchiveTeam/grab-site
A web crawler tool that outputs WARC files and provides a dashboard for managing crawls.
github/lightcrawler
A JavaScript tool that crawls a website and runs it through Google Lighthouse for performance, accessibility, and best practices analysis.
s0md3v/uro
A Python library that declutters URL lists for web crawling and penetration testing tasks.
Leon406/SubCrawler
A Kotlin-based open-source tool that automatically crawls and tests public network nodes for use in privacy-focused applications.
roach-php/core
A comprehensive web scraping toolkit for PHP developers, with capabilities for crawling, parsing, and extracting data from websites.
sethblack/python-seo-analyzer
A Python-based SEO analysis tool that crawls websites, counts words, and checks for technical SEO issues.
openwpm/OpenWPM
OpenWPM is a web privacy measurement framework written in Python that can be used to crawl and analyze websites.
zhuweiyou/weixin-game-helper
A collection of helper tools for various WeChat mini-games, with features like AI-powered game bots and data crawling.
chiphuyen/sotawhat
A Python tool that helps developers stay up-to-date with the latest AI research by summarizing Arxiv paper abstracts.
darbra/sperm
This is a collection of interesting reverse engineering articles worth checking out.
lorey/mlscraper
Effortlessly scrape data from websites using machine learning and HTML examples with this Python library.
mvdbos/php-spider
A configurable and extensible PHP web spider for crawling and extracting data from websites.
kgspider/crawler
A JavaScript web scraper repository focused on reverse engineering and advanced crawling techniques.
alongubkin/spider
A now-inactive JavaScript library for creating simple, unsurprising web crawling and scraping tools.
scrapinghub/frontera
A scalable frontier for web crawlers, focused on high-performance and scalable web crawling.
imgbot/Imgbot
An Azure Function to crawl GitHub repos and losslessly compress images, reducing file size while maintaining quality.
Darwin-lfl/langmanus
A community-driven AI automation framework that combines language models with specialized tools for tasks like web search, crawling, and Python code execution.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.