Skip to main content
Definition Website Computer and Technology

Stream Processing

Subject: Computer and Technology noun /ˈstriːm ˌprɑː.sɛs.ɪŋ/ 1 definition

  1. Definition (noun)

    Entry #303 · Updated

    Stream Processing is a computing paradigm that continuously ingests, evaluates, and transforms unbounded sequences of events in motion rather than waiting for static datasets to collect at rest. A stream processing engine executes persistent pipelines—typically structured as directed acyclic graphs (DAGs) of stateless operators such as map and filter alongside stateful operators like rolling aggregations and windowed joins—over partitioned event logs or message brokers. To preserve correctness despite network delays, node failures, and out-of-order records, engines decouple event time from processing time using watermarks, persist operator state through incremental checkpoints, and prevent overload via backpressure. Unlike batch processing, which computes over bounded datasets for maximum throughput and global context, stream processing optimizes for continuous sub-second latency and immediate actuation, making it ideal for real-time monitoring and event-driven automation while leaving massive historical backfills or multi-pass global analytics to batch or hybrid architectures.

    Common Forms

    • Native (Event-at-a-Time) Stream Processing: Evaluates each incoming record immediately upon arrival through continuously running pipelined operators (such as Apache Flink), achieving sub-second latencies and fine-grained per-event control.

    • Micro-Batch Stream Processing: Buffers arriving records over short intervals—typically tens to hundreds of milliseconds—and executes each slice as an atomic batch task (such as Spark Structured Streaming), simplifying task scheduling and recovery in exchange for slightly higher baseline latency.

    • Stateless Stream Processing: Transforms or routes each event in complete isolation—performing filtering, projection, or schema normalization—without maintaining cross-event context, enabling trivial horizontal scaling and minimal recovery overhead.

    • Stateful Stream Processing: Retains keyed operator state across continuous event sequences to compute sliding-window aggregations, stream-to-stream joins, and temporal pattern detection (Complex Event Processing), relying on local state backends and distributed checkpoints for exactness across failures.

    Example

    A payment gateway publishes authorization events such as {"card_id": "c-9042", "amount": 480.00, "ts": 1731951002} to an Apache Kafka topic payments.auth, which an Apache Flink job consumes continuously. Partitioned by card_id with a 5-minute sliding window and a 10-second event-time watermark, the pipeline tracks cumulative spend in RocksDB-backed operator state and emits an immediate fraud.alert event whenever a card's rolling total exceeds $2,000.