Vybe Guide
Sign Up

Explore Projects

Discover 170 open source projects

Active filters (1):
Search: crawler×
Clear all

Showing 1-20 of 170 projects

firecrawl/firecrawl

Convert websites into LLM-ready data with API for scraping, crawling, and structured data extraction

88.5K
Active
TypeScript
Web Scraping AI
Agents & Orchestration
TypeScript
#ai-scraping#web-crawler#llm-data

unclecode/crawl4ai

LLM-friendly web crawler & scraper for RAG, agents, and data pipelines

61.4K
Active
Python
RAG & Vector
CLI Tools
Python
#web-crawler#llm-ready#markdown

scrapy/scrapy

Scrapy is a fast, high-level web crawling and scraping framework for Python developers.

60.6K
Active
Python
Testing
Python
#web-scraping#crawling#python

NanmiCoder/MediaCrawler

Multi-platform social media crawler for content and comment scraping

44.9K
Active
Python
Testing
Web Scraping AI
Python
#web-scraping#social-media#data-mining

NaiboWang/EasySpider

Visual code-free web crawler/spider with GUI for data collection and automation

44.0K
Active
JavaScript
Testing
No-Code AI Platforms
#web-crawler#data-collection#gui

iawia002/lux

Fast video downloader in Go for various platforms

30.9K
Stable
Go
CLI Tools
Full-Stack Frameworks
Go
#video-downloader#go#cli

gocolly/colly

Go Colly - Elegant Scraper and Crawler Framework for Golang

25.1K
Active
Go
CLI Tools
Backend Frameworks
Go
#golang#scraper#crawler

D4Vinci/Scrapling

Powerful, flexible Python library for effortless web scraping with AI-powered features.

23.6K
Active
Python
Web Scraping
Backend Frameworks
Python
#web-scraping#automation#data-extraction

jhao104/proxy_pool

Python proxy pool for web scraping with Redis storage

23.2K
Stable
Python
Testing
Backend Frameworks
Python
#proxy#web-scraping#redis

ScrapeGraphAI/Scrapegraph-ai

AI-powered web scraping library for extracting data from websites and documents

22.9K
Active
Python
Web Scraping AI
RAG & Vector
Python
#ai-scraping#llm#rag

BuilderIO/gpt-crawler

Crawl websites to create custom GPTs from URLs

22.2K
Experimental
TypeScript
RAG & Vector
Web Scraping AI
TypeScript
#gpt#crawler#ai

apify/crawlee

Web scraping and browser automation library for Node.js

22.0K
Active
TypeScript
Browser Automation SDKs
Testing
Node.js
#web-scraping#browser-automation#nodejs

TecharoHQ/anubis

This Go library weighs the soul of incoming HTTP requests to stop AI crawlers from accessing your application.

17.4K
Active
Go
Security Research
#security#http#crawler-detection

binux/pyspider

A powerful web crawler system in Python for building custom web scraping solutions.

17.0K
Archived
Python
Backend Frameworks
Python
#web-crawler#web-scraping#python

Evil0ctal/Douyin_TikTok_Download_API

A high-performance async web scraping tool for extracting data from Douyin, TikTok, Bilibili and more.

16.5K
Stable
Python
API Frameworks
FastAPI
#api#async#scraper

projectdiscovery/katana

A highly customizable web crawler and spider framework for developers to build advanced crawling solutions.

15.7K
Active
Go
CLI Tools
Go
#crawler#spider-framework#web-spider

codelucas/newspaper

A Python library for extracting news articles, full-text, and metadata from websites.

15.0K
Stable
HTML
Backend Frameworks
Python
#web-scraping#news-extraction#data-extraction

shengqiangzhang/examples-of-web-crawlers

A collection of Python web crawler examples for various websites, suitable for beginners.

14.6K
Experimental
HTML
Backend Frameworks
Python
#web-scraping#crawlers#beginners

s0md3v/Photon

Incredibly fast and powerful OSINT crawler designed for information gathering and intelligence analysis.

12.7K
Stable
Python
API Clients & Testing
Python
#crawler#information-gathering#osint

crawlab-team/crawlab

A distributed web crawler admin platform for managing spiders in any language or framework.

12.2K
Stable
Go
Backend Frameworks
Go
#crawling#spider-management#distributed
2...9

Stay in the loop

Get weekly updates on trending AI coding tools and projects.