Explore Projects
Discover 39 open source projects
Showing 1-20 of 39 projects
pathwaycom/pathway
Python ETL framework for real-time analytics and LLM pipelines
apache/airflow
Apache Airflow for workflow orchestration
airbytehq/airbyte
Data integration platform for ELT pipelines from APIs, databases & files to databases, warehouses & lakes
apache/shardingsphere
Distributed SQL database middleware for sharding, scalability, and security
dagster-io/dagster
An open-source data orchestration platform for developing, running, and observing data pipelines and workflows.
apache/dolphinscheduler
Apache DolphinScheduler is a modern data orchestration platform for creating high-performance workflows with low-code.
Unstructured-IO/unstructured
Unstructured is an open-source ETL solution for transforming complex documents into structured data for language models.
debezium/debezium
An open-source framework for change data capture from various databases using Apache Kafka.
mage-ai/mage-ai
mage-ai is a Python-based platform for building, running, and managing data pipelines and integrating/transforming data.
snowplow/snowplow
A powerful customer data pipeline for collecting, processing, and analyzing user events and behavior.
apache/flink-cdc
Flink CDC is a streaming data integration tool that enables real-time data pipelines and change data capture.
datajuicer/data-juicer
A Python library for processing and analyzing data with foundation models and large language models.
fluvio-community/fluvio
Fluvio is an event stream processing engine for developers to build responsive data-intensive apps.
rudderlabs/rudder-server
Rudder Server is a privacy-focused, Segment-alternative customer data platform written in Go and React.
StructuredLabs/preswald
Preswald is a WASM packager for Python-based interactive data apps that can be run completely in-browser.
adilkhash/Data-Engineering-HowTo
A list of resources to learn Data Engineering from scratch
Netflix/maestro
Maestro is Netflix's workflow orchestrator for building data pipelines and batch processing workflows.
ucbepic/docetl
A system for agentic LLM-powered data processing and ETL workflows for unstructured data analysis.
superstreamlabs/memphis
Memphis.dev is a highly scalable and effortless data streaming platform
bruin-data/ingestr
ingestr is a CLI tool that seamlessly copies data between any databases with a single command.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.