Explore Projects
Discover 33 open source projects
Showing 1-20 of 33 projects
scrapy/scrapy
Scrapy is a fast, high-level web crawling and scraping framework for Python developers.
crawlab-team/crawlab
A distributed web crawler admin platform for managing spiders in any language or framework.
scrapinghub/portia
Portia is a visual scraping tool for Scrapy, a popular Python web scraping framework.
lining0806/PythonSpiderNotes
A Python-based web scraping library that provides code examples and techniques for various web scraping tasks.
rmax/scrapy-redis
A Redis-based distributed scraping library for the Scrapy web crawling framework.
SpiderClub/haipproxy
A high-availability distributed IP proxy pool powered by Scrapy and Redis for web crawling applications.
DropsDevopsOrg/ECommerceCrawlers
A collection of Python-based web crawlers for scraping data from various e-commerce and online platforms.
Boris-code/feapder
A powerful Python-based web scraping framework with features like browser rendering and data deduplication.
Gerapy/Gerapy
A distributed crawler management framework based on Scrapy, Scrapyd, Django, and Vue.js for web scraping.
my8100/scrapydweb
Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert for developers building with AI tools.
scrapy-plugins/scrapy-splash
A Scrapy plugin that integrates Splash, a headless web browser, for JavaScript rendering and scraping.
scrapy/scrapyd
A service daemon to run Scrapy spiders, a powerful web scraping library in Python.
DormyMo/SpiderKeeper
An open-source admin UI for the Scrapy web scraping framework, providing a dashboard for managing and monitoring spiders.
QianyanTech/Image-Downloader
A Python library for downloading images from Google, Bing, and Baidu.
librauee/Reptile
A comprehensive Python web scraping library covering a wide range of popular websites and platforms.
TheWebScrapingClub/webscraping-from-0-to-hero
A comprehensive resource for learning web scraping with Python, covering tools like Playwright, Scrapy, and Splash.
aivarsk/scrapy-proxies
A random proxy middleware for the Scrapy web scraping framework in Python.
scrapy/dirbot
A deprecated Python-based web scraping library for educational public web directories.
scrapy-plugins/scrapy-playwright
Scrapy-Playwright is a Python library that integrates the Playwright browser automation tool with the Scrapy web scraping framework.
eliasdabbas/advertools
A Python library for online marketing and web analytics tools, including SEO, advertising, and social media.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.