Skip to main content
Definition Website Computer and Technology

Backpropagation

Subject: Computer and Technology noun /ˌbæk.prɑː.pəˈɡeɪ.ʃən/ 1 definition

  1. Definition (noun)

    Entry #290 · Updated

    Backpropagation (short for backward propagation of errors) is the foundational reverse-mode automatic differentiation algorithm used to train artificial neural networks. During training, a forward pass first routes input data through the network to generate predictions and compute a scalar loss against target values, after which backpropagation traverses the computational graph in reverse. By recursively applying the calculus chain rule and reusing cached intermediate activations from the forward pass, it efficiently computes the gradient of the loss with respect to every weight and bias in a single backward sweep. Crucially, backpropagation is responsible strictly for calculating these gradients rather than adjusting weights itself, passing the computed derivatives to an optimizer such as stochastic gradient descent (SGD) or Adam to update the parameters.

    Common Forms

    • Standard (Feedforward) Backpropagation: Computes gradients across a directed acyclic computation graph in a single backward sweep, forming the core training mechanism for multilayer perceptrons, convolutional networks, and Transformers.

    • Backpropagation Through Time (BPTT): Unrolls recurrent neural networks across every step of an input sequence into an equivalent deep graph so gradients propagate backward through shared temporal weights, though long sequences increase memory use and risk vanishing or exploding gradients.

    • Truncated Backpropagation Through Time (TBPTT): Limits backward gradient flow to a fixed window of recent time steps rather than the full sequence length, bounding memory and compute overhead when training recurrent models on long sequences.

    • Gradient-Checkpointed Backpropagation: Discards selected intermediate forward activations to save device memory and recomputes them during the backward pass, trading extra arithmetic operations to train deeper networks or larger batches within fixed hardware limits.

    Example

    While training a convolutional image classifier in PyTorch on a mini-batch of 64 RGB images, the forward pass produces logits of shape [64, 10] and yields a cross-entropy loss of 1.42. Calling loss.backward() executes backpropagation through the computation graph to populate the .grad tensor of every convolutional kernel and linear weight before optimizer.step() updates those parameters.