Skip to main content
Definition Website Computer and Technology

Attention Mechanism

Subject: Computer and Technology noun /əˈtɛn.ʃən ˈmɛk.əˌnɪz.əm/ 1 definition

  1. Definition (noun)

    Entry #297 · Updated

    An Attention Mechanism is a differentiable neural network component that dynamically computes relevance weights over input elements to construct context-aware representations. In standard scaled dot-product attention, input representations are linearly projected into query, key, and value vectors; dot products between queries and keys are divided by the square root of the key dimension, optionally masked, and normalized with softmax to form a weighted sum over the values. By linking arbitrary sequence positions through direct, content-addressable paths, attention overcomes the fixed-state bottlenecks and sequential constraints of recurrent networks. Because dense pairwise scoring incurs O(n^2) time and memory complexity relative to sequence length n, long-context systems often mitigate hardware bottlenecks using key-value caching, memory-tiled kernels, or sparse and grouped-query approximations.

    Common Types

    • Self-Attention: Computes queries, keys, and values from a single input sequence so each token dynamically aggregates context from other tokens in that sequence, either bidirectionally in encoders or with causal masking in autoregressive decoders.

    • Cross-Attention: Draws queries from one sequence while taking keys and values from an external conditioning sequence, aligning representations between encoder outputs and decoder steps or across text, image, and audio modalities.

    • Multi-Head Attention (MHA): Projects queries, keys, and values into multiple independent lower-dimensional subspaces in parallel, enabling a single layer to capture syntactic, semantic, and positional relationships concurrently before concatenating the outputs.

    • Grouped-Query Attention (GQA): Shares a reduced set of key and value heads across clusters of query heads to compress inference-time key-value (KV) cache memory and bandwidth while retaining accuracy close to standard multi-head attention.

    Example

    During inference in a PyTorch-based 8-billion-parameter language model with 32 query heads, 8 key-value heads, and a head dimension of 128, a causal grouped-query attention layer calls torch.nn.functional.scaled_dot_product_attention over a 4,096-token prompt. For the final token, softmax-normalized query-key scores assign high weights—such as 0.42 to a function identifier at position 118 and 0.31 to its return type—aggregating their value vectors directly into the next-token representation.