ETL & Pipelines
Explore 310 open source projects in ETL & Pipelines
Showing 101-120 of 310 projects
sdv-dev/SDV
Generates synthetic tabular data for machine learning and AI applications
bruin-data/ingestr
ingestr is a CLI tool that seamlessly copies data between any databases with a single command.
databricks/koalas
Koalas is a pandas-like API for Apache Spark, enabling data scientists to work with big data using familiar pandas syntax.
WeBankFinTech/DataSphereStudio
DataSphereStudio is a one-stop data application development and management portal covering data exchange, analysis, and visualization.
lakesoul-io/LakeSoul
LakeSoul is a cloud-native, real-time Lakehouse framework for fast data ingestion and analytics on cloud storage.
apache/paimon
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark.
internetarchive/heritrix3
Heritrix is an open-source, extensible web crawler for archiving websites at scale.
pydata/pandas-datareader
A Python library for extracting data from a wide range of internet sources into a pandas DataFrame.
delta-io/delta-rs
A Rust library for interacting with Delta Lake, a data lake storage format, with Python bindings.
spark-notebook/spark-notebook
An interactive and reactive data science platform powered by Scala and Apache Spark.
blockchain-etl/ethereum-etl
Python scripts for extracting, transforming and loading Ethereum blockchain data into Google BigQuery.
gunnarmorling/awesome-opensource-data-engineering
An Awesome List of open-source data engineering projects for developers.
webdataset/webdataset
A high-performance I/O system for large deep learning problems with strong PyTorch support.
PeerDB-io/peerdb
Fast, cost-effective data replication tool from Postgres to data warehouses, queues, and storage
datafold/data-diff
A Python library for comparing data across databases, supporting various database engines.
alldatacenter/alldata
An open-source platform for building data-driven applications and AI-powered solutions with a focus on vibe coders.
x4nth055/pythoncode-tutorials
A collection of Python tutorials covering a wide range of topics from computer vision to network security.
apache/incubator-devlake
An open-source dev data platform to ingest, analyze, and visualize data from DevOps tools for engineering insights.
TobikoData/sqlmesh
Scalable and efficient data transformation framework with backwards compatibility for dbt.
huggingface/datatrove
A Python library that provides a set of customizable pipeline processing blocks for data processing tasks.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.