Transformer Architecture
Subject: Computer and Technology noun /trænsˈfɔːr.mər ˈɑːr.kɪˌtɛk.tʃər/ 1 definition
- deep-learning
- neural-networks
- machine-learning
- natural-language-processing
-
Definition (noun)
Entry #250 · UpdatedThe Transformer Architecture is a deep learning model design that relies on self-attention mechanisms and feed-forward layers to process sequential data without recurrence or convolutions. By computing attention weights across sequence tokens in parallel and using positional encodings to track order, it avoids step-by-step processing bottlenecks and captures long-range dependencies effectively, serving as the foundation for modern large language models and multimodal systems.
Common Types
Encoder-Only: Uses bidirectional self-attention across all input tokens simultaneously to produce contextual representations, making it suited for text classification, semantic search embeddings, and named-entity recognition.
Decoder-Only: Applies causal masking so each token attends only to preceding tokens, supporting scalable next-token prediction for conversational agents, code generation, and autoregressive reasoning.
Encoder-Decoder: Combines a bidirectional encoder with a cross-attention decoder to transform an input sequence into a target sequence, making it effective for machine translation, summarization, and speech transcription.
Example
When a developer prompts a decoder-only Transformer code assistant with a 3,000-token source file, positional encodings track line order while multi-head self-attention links a variable defined at the top of the file directly to a bug near the bottom. The model evaluates those relationships in parallel during the prefill pass and predicts the corrected patch token by token.