Skip to main content
Definition Website Computer and Technology

Quantization

Subject: Computer and Technology noun /ˌkwɑːn.təˈzeɪ.ʃən/ 1 definition

  1. Definition (noun)

    Entry #300 · Updated

    Quantization is the process of mapping values from a continuous range or high-precision numerical format onto a finite, discrete set of lower-precision representation levels. It partitions an input domain into discrete intervals—often parameterized by step sizes, scale factors, and optional zero-point offsets—and rounds each analog sample, transform coefficient, or floating-point parameter to its nearest representable level, such as an 8-bit or 4-bit integer. By encoding numbers with fewer bits, quantization shrinks memory footprints, lowers data-transfer bandwidth, and enables hardware to execute fast integer or mixed-precision instructions across digital signal processors, GPUs, and neural accelerators. Because rounding discards fine-grained detail irreversibly, it introduces quantization error—a fundamental trade-off between compression ratio or throughput and reconstruction fidelity or model accuracy that separates lossy quantization from lossless encoding.

    Common Forms

    • Uniform Scalar Quantization: Maps individual values onto evenly spaced intervals using a constant step size and optional zero-point offset, enabling simple, hardware-friendly integer arithmetic in PCM audio, JPEG DCT matrices, and INT8 inference.

    • Non-Uniform Quantization: Allocates narrower intervals in high-density regions and wider intervals in sparse tails—such as logarithmic μ-law companding or NormalFloat (NF4) lookup tables—to preserve precision on skewed distributions without adding bits.

    • Vector Quantization (VQ): Groups correlated scalars into multi-dimensional vectors and replaces each vector with the index of its closest centroid in a learned codebook, raising compression efficiency in speech codecs, VQ-VAE tokenizers, and product-quantized vector search.

    • Post-Training and Quantization-Aware Variants: Calibrates neural network scale factors after training from representative data or runtime ranges (Post-Training Quantization, PTQ), or simulates rounding inside forward passes via straight-through estimators during optimization (Quantization-Aware Training, QAT) to retain accuracy at low bit widths.

    Example

    To meet a 15 ms latency target on CPU inference servers, a search service converts a 440 MB FP32 transformer reranker to INT8 via ONNX Runtime's quantize_dynamic("reranker.onnx", "reranker_int8.onnx", weight_type=QuantType.QInt8), shrinking the model file to roughly 112 MB. By mapping floating-point tensor ranges onto signed [-128, 127] integer levels using per-channel scale factors, the runtime executes matrix multiplications with AVX-512 VNNI integer dot-product instructions rather than 32-bit float units, halving inference latency with minimal ranking degradation.