Skip to main content
Definition Website Computer and Technology

Batch Processing

Subject: Computer and Technology noun /ˈbætʃ ˌprɑː.sɛs.ɪŋ/ 1 definition

  1. Definition (noun)

    Entry #305 · Updated

    Batch Processing is a computing model in which a bounded collection of data records or computational tasks is accumulated into a discrete group—known as a batch—and executed sequentially or in parallel without continuous human intervention. Rather than responding to individual inputs as they arrive, a job scheduler or orchestration engine dispatches each queued workload once a predefined schedule, resource window, or data-volume threshold is met, reading inputs from durable storage and writing finalized outputs upon completion. By amortizing startup, I/O, and context-switching overhead across large volumes of items, batch execution maximizes hardware utilization and sustained throughput while enabling straightforward checkpointing and deterministic job retries. Because it introduces intentional queuing latency between data arrival and result availability, batch processing is best suited to periodic workloads—such as payroll runs, billing reconciliation, and analytical reporting—rather than real-time interactive applications or low-latency stream processing.

    Common Forms

    • Scheduled Batch Processing: Executes jobs automatically on fixed time intervals—such as hourly cron tasks, nightly ETL pipelines, or monthly billing runs—aligning heavy computation with predictable reporting cycles and off-peak capacity.

    • Threshold-Triggered Batch Processing: Accumulates incoming records, database writes, or API payloads in a buffer until a target item count or byte limit is reached, then flushes the entire group together to amortize I/O and network round-trip overhead.

    • Distributed Batch Processing: Partitions massive bounded datasets across clusters of compute nodes using frameworks such as MapReduce or Apache Spark, executing parallel map, shuffle, and reduce stages to process terabyte-to-petabyte workloads.

    • Micro-Batch Processing: Slices continuous event streams into short, fixed-duration or count-bounded chunks—typically spanning milliseconds to a few seconds—to achieve near-real-time latency while retaining batch fault tolerance and deterministic replay.

    Example

    At 02:00 UTC each night, an Apache Airflow DAG triggers an Apache Spark batch job that reads 1.4 TB of raw clickstream_events Parquet files from Amazon S3 for the prior UTC date. Across 32 worker executors, the job deduplicates session IDs, aggregates daily conversion funnels, and commits the summarized partitions to a Snowflake data warehouse table in a single atomic MERGE transaction.