Explore Projects
Discover 96 open source projects
Showing 1-20 of 96 projects
firecrawl/firecrawl
Convert websites into LLM-ready data with API for scraping, crawling, and structured data extraction
scrapy/scrapy
Scrapy is a fast, high-level web crawling and scraping framework for Python developers.
gocolly/colly
Go Colly - Elegant Scraper and Crawler Framework for Golang
bytedance/deer-flow
DeerFlow is a super agent harness for orchestrating sub-agents, memory, and sandboxes in deep research workflows.
D4Vinci/Scrapling
Powerful, flexible Python library for effortless web scraping with AI-powered features.
BuilderIO/gpt-crawler
Crawl websites to create custom GPTs from URLs
apify/crawlee
Web scraping and browser automation library for Node.js
projectdiscovery/katana
A highly customizable web crawler and spider framework for developers to build advanced crawling solutions.
codelucas/newspaper
A Python library for extracting news articles, full-text, and metadata from websites.
waditu/tushare
A Python library for crawling historical data of China stocks.
crawlab-team/crawlab
A distributed web crawler admin platform for managing spiders in any language or framework.
ssssssss-team/spider-flow
A no-code web crawler platform that allows developers to define crawling workflows visually without writing code.
dataabc/weiboSpider
Crawls and scrapes Weibo data using Python.
kangvcar/InfoSpider
INFO-SPIDER is an open-source web scraping toolkit that helps users retrieve data from various sources like email, e-commerce, and social platforms.
apify/crawlee-python
Crawlee is a powerful web scraping and browser automation library for Python to build reliable crawlers.
lorien/awesome-web-scraping
A comprehensive list of libraries, tools, and APIs for web scraping and data processing.
andeya/pholcus
Pholcus is a high-concurrency web crawler software written in Go for developers needing a powerful, distributed crawling solution.
luyishisi/Anti-Anti-Spider
Anti-Anti-Spider is a Python library that helps developers bypass anti-crawling measures on websites to collect data.
go-rod/rod
A Go library for automating and scraping websites using the Chrome DevTools Protocol.
wzdnzd/aggregator
A Python-based platform for aggregating and crawling proxy servers, useful for building proxy-reliant applications.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.