Gradient Descent
Subject: Computer and Technology noun /ˈɡreɪ.di.ənt dɪˈsɛnt/ 1 definition
- machine-learning
- deep-learning
- optimization-algorithms
- numerical-computing
-
Definition (noun)
Entry #285 · UpdatedGradient Descent is an iterative first-order optimization algorithm used to find a local or global minimum of a differentiable objective function, most commonly a loss function in machine learning. Starting from an initial parameter vector, each step computes the gradient—the vector of partial derivatives pointing in the direction of steepest ascent—and subtracts a fraction of that gradient, scaled by a step-size hyperparameter called the learning rate, from the current parameters. Because it relies exclusively on first-order derivatives rather than forming second-order Hessian matrices, the algorithm scales efficiently to high-dimensional models with millions or billions of parameters. Its effectiveness is bounded by differentiability and step-size sensitivity, as overly large learning rates cause divergence, overly small ones slow progress, and non-convex loss landscapes can trap iterates near saddle points or suboptimal local minima.
Common Forms
Batch Gradient Descent: Computes the exact gradient across the entire training dataset before performing a single parameter update, producing smooth, deterministic convergence at the cost of high memory usage and slow iterations on large datasets.
Stochastic Gradient Descent (SGD): Updates parameters using the gradient estimated from a single randomly sampled training example per step, introducing noisy updates that cut per-step cost and help escape shallow local minima.
Mini-Batch Gradient Descent: Balances computational efficiency and gradient stability by computing updates over small fixed-size subsets of data (such as 32 to 512 samples), enabling vectorized GPU and TPU parallelization.
Momentum and Adaptive Variants: Extends standard updates with running averages of past gradients or squared gradients—such as classical momentum, RMSprop, and Adam—to dampen oscillations in steep ravines and automatically scale per-parameter learning rates.
Example
While training a 10-class ResNet-18 image classifier in PyTorch with
torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)over mini-batches of 64 images, each step evaluates cross-entropy loss, callsloss.backward()to compute parameter gradients, and runsoptimizer.step()to shift weights opposite the gradient. Over 30 epochs on a GPU, these iterative updates steadily lower validation loss from2.31to0.18as the model's predictions converge toward the ground-truth labels.