ETL & Pipelines
Explore 310 open source projects in ETL & Pipelines
Showing 81-100 of 310 projects
apache/streampark
Easy-to-use streaming application development framework and operation platform for building ETL pipelines.
StructuredLabs/preswald
Preswald is a WASM packager for Python-based interactive data apps that can be run completely in-browser.
run-llama/llama_cloud_services
A set of TypeScript-based cloud services and utilities for processing and extracting structured data from various document formats.
zendesk/maxwell
Maxwell's daemon, a MySQL-to-JSON Kafka producer for building real-time data pipelines.
attardi/wikiextractor
A Python tool for extracting plain text from Wikipedia dumps, useful for natural language processing tasks.
adilkhash/Data-Engineering-HowTo
A list of resources to learn Data Engineering from scratch
quadratichq/quadratic
A spreadsheet tool with AI capabilities for data analysis, engineering, and visualization.
jghoman/awesome-apache-airflow
Curated list of resources about Apache Airflow, a popular workflow management platform.
multiprocessio/dsq
A command-line tool for running SQL queries against various data formats like JSON, CSV, Excel, and Parquet.
puckel/docker-airflow
A Docker-based Apache Airflow platform for building and managing data pipelines and workflows.
Netflix/maestro
Maestro is Netflix's workflow orchestrator for building data pipelines and batch processing workflows.
atlanhq/camelot
Camelot is a Python library for extracting tables from PDF files, making it easier for developers to work with PDF data.
DataLinkDC/dinky
Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.
ucbepic/docetl
A system for agentic LLM-powered data processing and ETL workflows for unstructured data analysis.
mrdbourke/zero-to-mastery-ml
A comprehensive machine learning and data science course with Jupyter Notebook materials.
ploomber/ploomber
Ploomber is a fast and versatile tool for building and deploying data pipelines that can be used with a variety of AI and ML tools.
awslabs/deequ
Deequ is a Scala library for defining "unit tests for data" to measure data quality in large datasets.
noflo/noflo
Flow-based programming framework for building complex JavaScript applications and services.
dathere/qsv
Blazing-fast data wrangling toolkit for AI and data engineering workflows
xyflow/awesome-node-based-uis
A curated list of resources for creating node-based UI editors and visual programming tools.
Stay in the loop
Get weekly updates on trending AI coding tools and projects.