ETL & Pipelines
Explore 310 open source projects in ETL & Pipelines
Showing 221-240 of 310 projects
OpnTec/parliament-scraper
A Python scraper for public data from the EU and other parliament websites.
AI4Finance-Foundation/FinNLP
Democratizing internet-scale financial data for developers through natural language processing.
wgzhao/Addax
A fast and versatile ETL tool that can transfer data between RDBMS and NoSQL databases seamlessly
zstmfhy/zlibrary-to-notebooklm
A Python script that automatically downloads books from Z-Library and uploads them to Google NotebookLM.
toluaina/pgsync
A Python library that syncs data from Postgres to Elasticsearch/OpenSearch, enabling real-time data pipelines.
damklis/DataEngineeringProject
An end-to-end data engineering project example showcasing tools and technologies for building data pipelines.
databricks/LearningSparkV2
This is a book that teaches how to use Apache Spark for lightning-fast data analytics.
lorey/mlscraper
Effortlessly scrape data from websites using machine learning and HTML examples with this Python library.
PKUJohnson/OpenData
An open-source financial data extraction tool that allows easy API access to web scrape data from various websites.
quiltdata/quilt
Quilt is a data mesh for connecting people with actionable data, built with TypeScript.
gtoonstra/etl-with-airflow
This repository provides best practices and examples for building ETL (Extract, Transform, Load) pipelines using Apache Airflow.
xisuo67/XHS-Spider
A web scraping and data collection tool for the Chinese social media platform Xiaohongshu (Little Red Book).
amphi-ai/amphi-etl
A visual data preparation tool powered by Python, designed for data analysis and ETL tasks.
apache/hop
Hop is a flexible and extensible open-source data integration platform for building and orchestrating ETL and streaming pipelines.
PDAL/PDAL
PDAL is a C++ library for processing point cloud data, similar to GDAL for raster data.
ropensci/drake
An R-focused pipeline toolkit for reproducibility and high-performance computing.
rwynn/monstache
A Go daemon that syncs MongoDB to Elasticsearch in real-time for search-powered applications.
singer-io/getting-started
A getting started guide to Singer, a data integration framework for ETL and data analysis.
alan-turing-institute/CleverCSV
A Python package for handling messy CSV files with improved dialect detection and a command-line interface.
PatMartin/Dex
Dex is a powerful data visualization tool that enables data exploration and publishing of web visualizations.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.