Explore Projects
Discover 170 open source projects
Showing 121-140 of 170 projects
ArchiveTeam/grab-site
A web crawler tool that outputs WARC files and provides a dashboard for managing crawls.
github/lightcrawler
A JavaScript tool that crawls a website and runs it through Google Lighthouse for performance, accessibility, and best practices analysis.
oxylabs/how-to-scrape-google-images
Automated Google image scraper that retrieves and parses image data using HTTP requests.
danielmiessler/RobotsDisallowed
A curated list of commonly disallowed robots.txt directories, useful for web crawlers and SEO.
Autumn-27/ScopeSentry
ScopeSentry is a comprehensive security tool for mapping cyberspace, enumerating subdomains, scanning ports, and identifying vulnerabilities.
keenwon/antcolony
A Node.js-based web scraper and crawler for finding and downloading torrent files.
dadoonet/fscrawler
An Elasticsearch-powered file system crawler that can index content from various file formats.
openwpm/OpenWPM
OpenWPM is a web privacy measurement framework written in Python that can be used to crawl and analyze websites.
darbra/sperm
This is a collection of interesting reverse engineering articles worth checking out.
LeonardoCardoso/SwiftLinkPreview
A Swift library that generates previews from URLs, extracting titles, images, and relevant text.
lorey/mlscraper
Effortlessly scrape data from websites using machine learning and HTML examples with this Python library.
amirgamil/apollo
A personal search engine and web crawler for developers to explore their digital footprint.
monperrus/crawler-user-agents
A Go library that provides a database of syntactic patterns of HTTP user-agents used by bots/crawlers/scrapers.
felipecsl/wombat
Lightweight Ruby web crawler/scraper with an elegant DSL to extract structured data from pages.
sec-edgar/sec-edgar
Downloads periodic reports, filings and forms from EDGAR database.
eliasdabbas/advertools
A Python library for online marketing and web analytics tools, including SEO, advertising, and social media.
xisuo67/XHS-Spider
A web scraping and data collection tool for the Chinese social media platform Xiaohongshu (Little Red Book).
kgspider/crawler
A JavaScript web scraper repository focused on reverse engineering and advanced crawling techniques.
scrapinghub/frontera
A scalable frontier for web crawlers, focused on high-performance and scalable web crawling.
adyzng/jd-autobuy
A Python web scraper for automatically logging in and purchasing items on JD.com.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.