Explore Projects
Discover 17 open source projects
Showing 1-17 of 17 projects
firecrawl/firecrawl
Convert websites into LLM-ready data with API for scraping, crawling, and structured data extraction
ScrapeGraphAI/Scrapegraph-ai
AI-powered web scraping library for extracting data from websites and documents
apify/crawlee
Web scraping and browser automation library for Node.js
crawlab-team/crawlab
A distributed web crawler admin platform for managing spiders in any language or framework.
ssssssss-team/spider-flow
A no-code web crawler platform that allows developers to define crawling workflows visually without writing code.
apify/crawlee-python
Crawlee is a powerful web scraping and browser automation library for Python to build reliable crawlers.
BruceDone/awesome-crawler
A comprehensive collection of web crawlers and scrapers in various programming languages.
adithya-s-k/omniparse
A Python library for ingesting, parsing, and optimizing any data format for enhanced compatibility with GenAI frameworks.
firecrawl/firecrawl-mcp-server
Firecrawl MCP Server adds powerful web scraping and search capabilities to AI language models like Cursor and Claude.
jasonxtn/Argus
A comprehensive toolkit for information gathering and reconnaissance, including OSINT, web crawling, and more.
apache/nutch
Apache Nutch is an extensible and scalable web crawler for building search engines and data mining applications.
sjdirect/abot
A cross-platform, fast, and flexible C# web crawler framework for developers building crawlers and spiders.
xianhu/PSpider
A simple and easy-to-use Python web scraping framework with support for multi-threading and proxies.
MarginaliaSearch/MarginaliaSearch
An internet search engine focused on indexing the small, old, and weird parts of the web.
gildas-lormeau/single-file-cli
CLI tool for saving a complete web page as a single HTML file, useful for web archiving and scraping.
JustinBeckwith/linkinator
A TypeScript-based tool for finding and fixing broken links in websites, documentation, and local files.
omrilotan/isbot
A TypeScript library to detect bots, crawlers, and spiders based on their user agent string.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.