Quantization
Subject: Computer and Technology noun /ˌkwɑːn.təˈzeɪ.ʃən/ 1 definition
- machine-learning
- signal-processing
- data-compression
- computer-science
-
Definition (noun)
Entry #300 · UpdatedQuantization is the process of mapping values from a continuous range or high-precision numerical format onto a finite, discrete set of lower-precision representation levels. It partitions an input domain into discrete intervals—often parameterized by step sizes, scale factors, and optional zero-point offsets—and rounds each analog sample, transform coefficient, or floating-point parameter to its nearest representable level, such as an 8-bit or 4-bit integer. By encoding numbers with fewer bits, quantization shrinks memory footprints, lowers data-transfer bandwidth, and enables hardware to execute fast integer or mixed-precision instructions across digital signal processors, GPUs, and neural accelerators. Because rounding discards fine-grained detail irreversibly, it introduces quantization error—a fundamental trade-off between compression ratio or throughput and reconstruction fidelity or model accuracy that separates lossy quantization from lossless encoding.
Common Forms
Uniform Scalar Quantization: Maps individual values onto evenly spaced intervals using a constant step size and optional zero-point offset, enabling simple, hardware-friendly integer arithmetic in PCM audio, JPEG DCT matrices, and INT8 inference.
Non-Uniform Quantization: Allocates narrower intervals in high-density regions and wider intervals in sparse tails—such as logarithmic μ-law companding or NormalFloat (NF4) lookup tables—to preserve precision on skewed distributions without adding bits.
Vector Quantization (VQ): Groups correlated scalars into multi-dimensional vectors and replaces each vector with the index of its closest centroid in a learned codebook, raising compression efficiency in speech codecs, VQ-VAE tokenizers, and product-quantized vector search.
Post-Training and Quantization-Aware Variants: Calibrates neural network scale factors after training from representative data or runtime ranges (Post-Training Quantization, PTQ), or simulates rounding inside forward passes via straight-through estimators during optimization (Quantization-Aware Training, QAT) to retain accuracy at low bit widths.
Example
To meet a 15 ms latency target on CPU inference servers, a search service converts a 440 MB
FP32transformer reranker toINT8via ONNX Runtime'squantize_dynamic("reranker.onnx", "reranker_int8.onnx", weight_type=QuantType.QInt8), shrinking the model file to roughly 112 MB. By mapping floating-point tensor ranges onto signed[-128, 127]integer levels using per-channel scale factors, the runtime executes matrix multiplications withAVX-512 VNNIinteger dot-product instructions rather than 32-bit float units, halving inference latency with minimal ranking degradation.