Data Pipeline
Subject: Computer and Technology noun /ˈdeɪ.tə ˈpaɪp.laɪn/ 1 definition
- data-engineering
- data-integration
- data-processing
- distributed-systems
-
Definition (noun)
Entry #295 · UpdatedA Data Pipeline is an automated software architecture that moves data through connected processing stages from one or more source systems—such as transactional databases, application logs, or IoT sensors—to downstream destinations like data warehouses, data lakes, or machine learning services. Each stage performs a discrete operation—including ingestion, schema validation, filtering, enrichment, aggregation, or format conversion—passing intermediate results along a directed execution graph managed by workflow orchestrators, message brokers, or distributed compute engines. Production pipelines rely on checkpointing, idempotent writes, retry policies, and dead-letter queues to tolerate partial failures and preserve data integrity across scheduled runs or continuous event streams. By decoupling data collection and preparation from downstream consumption, a data pipeline provides reliable, repeatable delivery without forcing analytical tools or external applications to query operational sources directly.
Common Types
Batch Pipeline: Collects and processes bounded datasets on scheduled intervals (such as hourly or nightly jobs), maximizing throughput and compute efficiency for periodic reporting, billing, and model training.
Streaming Pipeline: Processes unbounded event flows continuously as records arrive—typically using message brokers and stateful stream engines—to enable sub-second alerting, fraud scoring, and live operational dashboards.
ETL (Extract, Transform, Load) Pipeline: Cleans, masks, and normalizes extracted records on an intermediate compute engine before writing them to the target store, ensuring sensitive fields are redacted and schemas conform prior to persistence.
ELT (Extract, Load, Transform) Pipeline: Lands raw extracted data directly into a scalable cloud warehouse or lakehouse and executes transformations in place using SQL or distributed compute, preserving raw history and adapting easily to evolving analytics requirements.
Example
A retail analytics platform streams checkout events serialized in Apache Avro over HTTPS into an Apache Kafka topic (
orders.completed), where an Apache Flink job validates schemas against a Confluent Schema Registry, redactscard_last4fields, and aggregates revenue in 5-minute tumbling windows. Using two-phase commit checkpoints, the pipeline writes partitioned Apache Parquet files to Amazon S3 (s3://lake-prod/orders/dt=2026-11-18/) and triggers adbt runtask in Apache Airflow to refresh downstream Snowflake reporting tables.