Skip to main content
Definition Website Computer and Technology

Transformer Architecture

Subject: Computer and Technology noun /trænsˈfɔːr.mər ˈɑːr.kɪˌtɛk.tʃər/ 1 definition

  1. Definition (noun)

    Entry #250 · Updated

    The Transformer Architecture is a deep learning model design that relies on self-attention mechanisms and feed-forward layers to process sequential data without recurrence or convolutions. By computing attention weights across sequence tokens in parallel and using positional encodings to track order, it avoids step-by-step processing bottlenecks and captures long-range dependencies effectively, serving as the foundation for modern large language models and multimodal systems.

    Common Types

    • Encoder-Only: Uses bidirectional self-attention across all input tokens simultaneously to produce contextual representations, making it suited for text classification, semantic search embeddings, and named-entity recognition.

    • Decoder-Only: Applies causal masking so each token attends only to preceding tokens, supporting scalable next-token prediction for conversational agents, code generation, and autoregressive reasoning.

    • Encoder-Decoder: Combines a bidirectional encoder with a cross-attention decoder to transform an input sequence into a target sequence, making it effective for machine translation, summarization, and speech transcription.

    Example

    When a developer prompts a decoder-only Transformer code assistant with a 3,000-token source file, positional encodings track line order while multi-head self-attention links a variable defined at the top of the file directly to a bug near the bottom. The model evaluates those relationships in parallel during the prefill pass and predicts the corrected patch token by token.