Skip to main content
Definition Website Computer and Technology

Gradient Descent

Subject: Computer and Technology noun /ˈɡreɪ.di.ənt dɪˈsɛnt/ 1 definition

  1. Definition (noun)

    Entry #285 · Updated

    Gradient Descent is an iterative first-order optimization algorithm used to find a local or global minimum of a differentiable objective function, most commonly a loss function in machine learning. Starting from an initial parameter vector, each step computes the gradient—the vector of partial derivatives pointing in the direction of steepest ascent—and subtracts a fraction of that gradient, scaled by a step-size hyperparameter called the learning rate, from the current parameters. Because it relies exclusively on first-order derivatives rather than forming second-order Hessian matrices, the algorithm scales efficiently to high-dimensional models with millions or billions of parameters. Its effectiveness is bounded by differentiability and step-size sensitivity, as overly large learning rates cause divergence, overly small ones slow progress, and non-convex loss landscapes can trap iterates near saddle points or suboptimal local minima.

    Common Forms

    • Batch Gradient Descent: Computes the exact gradient across the entire training dataset before performing a single parameter update, producing smooth, deterministic convergence at the cost of high memory usage and slow iterations on large datasets.

    • Stochastic Gradient Descent (SGD): Updates parameters using the gradient estimated from a single randomly sampled training example per step, introducing noisy updates that cut per-step cost and help escape shallow local minima.

    • Mini-Batch Gradient Descent: Balances computational efficiency and gradient stability by computing updates over small fixed-size subsets of data (such as 32 to 512 samples), enabling vectorized GPU and TPU parallelization.

    • Momentum and Adaptive Variants: Extends standard updates with running averages of past gradients or squared gradients—such as classical momentum, RMSprop, and Adam—to dampen oscillations in steep ravines and automatically scale per-parameter learning rates.

    Example

    While training a 10-class ResNet-18 image classifier in PyTorch with torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9) over mini-batches of 64 images, each step evaluates cross-entropy loss, calls loss.backward() to compute parameter gradients, and runs optimizer.step() to shift weights opposite the gradient. Over 30 epochs on a GPU, these iterative updates steadily lower validation loss from 2.31 to 0.18 as the model's predictions converge toward the ground-truth labels.