Attention Mechanism
Subject: Computer and Technology noun /əˈtɛn.ʃən ˈmɛk.əˌnɪz.əm/ 1 definition
- artificial-intelligence
- deep-learning
- machine-learning
- neural-networks
-
Definition (noun)
Entry #297 · UpdatedAn Attention Mechanism is a differentiable neural network component that dynamically computes relevance weights over input elements to construct context-aware representations. In standard scaled dot-product attention, input representations are linearly projected into query, key, and value vectors; dot products between queries and keys are divided by the square root of the key dimension, optionally masked, and normalized with softmax to form a weighted sum over the values. By linking arbitrary sequence positions through direct, content-addressable paths, attention overcomes the fixed-state bottlenecks and sequential constraints of recurrent networks. Because dense pairwise scoring incurs
O(n^2)time and memory complexity relative to sequence lengthn, long-context systems often mitigate hardware bottlenecks using key-value caching, memory-tiled kernels, or sparse and grouped-query approximations.Common Types
Self-Attention: Computes queries, keys, and values from a single input sequence so each token dynamically aggregates context from other tokens in that sequence, either bidirectionally in encoders or with causal masking in autoregressive decoders.
Cross-Attention: Draws queries from one sequence while taking keys and values from an external conditioning sequence, aligning representations between encoder outputs and decoder steps or across text, image, and audio modalities.
Multi-Head Attention (MHA): Projects queries, keys, and values into multiple independent lower-dimensional subspaces in parallel, enabling a single layer to capture syntactic, semantic, and positional relationships concurrently before concatenating the outputs.
Grouped-Query Attention (GQA): Shares a reduced set of key and value heads across clusters of query heads to compress inference-time key-value (KV) cache memory and bandwidth while retaining accuracy close to standard multi-head attention.
Example
During inference in a PyTorch-based 8-billion-parameter language model with 32 query heads, 8 key-value heads, and a head dimension of 128, a causal grouped-query attention layer calls
torch.nn.functional.scaled_dot_product_attentionover a 4,096-token prompt. For the final token, softmax-normalized query-key scores assign high weights—such as0.42to a function identifier at position118and0.31to its return type—aggregating their value vectors directly into the next-token representation.