Skip to main content
Definition Website Computer and Technology

Knowledge Distillation

Subject: Computer and Technology noun /ˈnɑː.lɪdʒ ˌdɪs.təˈleɪ.ʃən/ 1 definition

  1. Definition (noun)

    Entry #294 · Updated

    Knowledge Distillation is a machine learning model compression technique in which a compact "student" network is trained to reproduce the predictive behavior and internal representations of a larger, higher-capacity "teacher" model or ensemble. Instead of supervising the student solely with discrete one-hot ground-truth labels, training minimizes a composite objective—typically combining standard cross-entropy with Kullback-Leibler (KL) divergence—against the teacher's temperature-scaled softmax probabilities ("soft targets"), exposing rich inter-class similarities that hard labels discard. By transferring nuanced decision boundaries into smaller architectures, the process preserves much of the teacher's generalization accuracy while substantially lowering parameter count, memory footprint, and inference latency. Unlike quantization or pruning, which compress a single trained network in place, distillation bridges distinct model architectures, though student quality remains bounded by teacher errors, architecture capacity gaps, and teacher supervision costs.

    Common Forms

    • Response-Based Distillation: Trains the student to match the teacher's final output layer—such as temperature-scaled logits, class probabilities, or generated text sequences from black-box models—directly aligning predictive behavior.

    • Feature-Based Distillation: Matches intermediate hidden-layer activations, attention maps, or embedding projections between teacher and student networks, giving smaller models richer structural guidance across internal layers.

    • Relation-Based Distillation: Transfers geometric relationships across multiple training examples or pairs of network layers, such as pairwise similarity matrices or feature-map Gram matrices, rather than matching isolated single-sample outputs.

    • Online and Self-Distillation: Eliminates a separate pre-trained teacher phase by either training peer networks simultaneously to teach one another or distilling deeper layers and earlier training checkpoints back into shallower branches of the same model.

    Example

    To meet a 15 ms latency budget in a customer-support intent-routing service, an engineering team distills a fine-tuned 12-layer BERT-base teacher (110M parameters) into a 6-layer transformer student (66M parameters). During PyTorch training, they apply a softmax temperature of T = 2.0 and combine T^2-scaled KL divergence (torch.nn.functional.kl_div) weighted at 0.7 with ground-truth cross-entropy weighted at 0.3, cutting CPU inference latency from 28 ms to 13 ms while retaining 97.8% of the teacher's F1 score.