Backpropagation
Subject: Computer and Technology noun /ˌbæk.prɑː.pəˈɡeɪ.ʃən/ 1 definition
- machine-learning
- deep-learning
- neural-networks
- algorithms
-
Definition (noun)
Entry #290 · UpdatedBackpropagation (short for backward propagation of errors) is the foundational reverse-mode automatic differentiation algorithm used to train artificial neural networks. During training, a forward pass first routes input data through the network to generate predictions and compute a scalar loss against target values, after which backpropagation traverses the computational graph in reverse. By recursively applying the calculus chain rule and reusing cached intermediate activations from the forward pass, it efficiently computes the gradient of the loss with respect to every weight and bias in a single backward sweep. Crucially, backpropagation is responsible strictly for calculating these gradients rather than adjusting weights itself, passing the computed derivatives to an optimizer such as stochastic gradient descent (SGD) or Adam to update the parameters.
Common Forms
Standard (Feedforward) Backpropagation: Computes gradients across a directed acyclic computation graph in a single backward sweep, forming the core training mechanism for multilayer perceptrons, convolutional networks, and Transformers.
Backpropagation Through Time (BPTT): Unrolls recurrent neural networks across every step of an input sequence into an equivalent deep graph so gradients propagate backward through shared temporal weights, though long sequences increase memory use and risk vanishing or exploding gradients.
Truncated Backpropagation Through Time (TBPTT): Limits backward gradient flow to a fixed window of recent time steps rather than the full sequence length, bounding memory and compute overhead when training recurrent models on long sequences.
Gradient-Checkpointed Backpropagation: Discards selected intermediate forward activations to save device memory and recomputes them during the backward pass, trading extra arithmetic operations to train deeper networks or larger batches within fixed hardware limits.
Example
While training a convolutional image classifier in PyTorch on a mini-batch of 64 RGB images, the forward pass produces logits of shape
[64, 10]and yields a cross-entropy loss of1.42. Callingloss.backward()executes backpropagation through the computation graph to populate the.gradtensor of every convolutional kernel and linear weight beforeoptimizer.step()updates those parameters.