Word Embedding
Subject: Computer and Technology noun /ˈwɜːrd ɪmˈbɛd.ɪŋ/ 1 definition
- natural-language-processing
- machine-learning
- deep-learning
- information-retrieval
-
Definition (noun)
Entry #301 · UpdatedA Word Embedding is a dense, real-valued vector representation of a word or subword token in a continuous, low-dimensional geometric space, typically spanning tens to a few thousand dimensions. Unlike sparse one-hot encodings—which assign each vocabulary entry an orthogonal index without capturing term relatedness—embeddings are learned from large text corpora under the distributional hypothesis so that words sharing similar contexts map to nearby coordinates. Within this continuous space, operations such as cosine similarity and dot product measure semantic and syntactic relatedness, while linear vector offsets frequently encode regular linguistic relationships like verb tense or singular-to-plural inflection. By bridging discrete symbolic text and differentiable neural computation, word embeddings enable language models to generalize across unseen phrasing, constrained primarily by vocabulary coverage, domain shift, and whether vectors remain fixed or adapt to sentence context.
Common Types
Static Predictive Embeddings: Learn one fixed vector per vocabulary word by training shallow neural models over local sliding windows—such as Word2Vec's Continuous Bag-of-Words (CBOW) and Skip-gram architectures—to predict target words from neighbors or vice versa.
Count-Based Co-Occurrence Embeddings: Construct static word vectors by factorizing global word-to-word co-occurrence matrices—exemplified by GloVe—training vector dot products to approximate log co-occurrence probabilities across an entire corpus.
Subword Embeddings: Compose each word vector by summing representations of its constituent character n-grams—as pioneered by fastText—enabling models to synthesize meaningful vectors for rare, inflected, misspelled, or out-of-vocabulary terms.
Contextualized Embeddings: Generate context-dependent token vectors at inference time using deep sequence encoders such as ELMo or BERT, assigning distinct representations to identical spellings—such as
"bank"in"river bank"versus"bank account"—to resolve lexical ambiguity.
Example
An e-commerce search service loads a 300-dimensional Word2Vec model via Python's
gensim.models.KeyedVectorsto expand user queries with semantic synonyms before dispatching them to Elasticsearch. When a shopper searches for"laptop", the service invokeswv.most_similar("laptop", topn=3)to retrieve"notebook"(0.82),"ultrabook"(0.77), and"chromebook"(0.73) by cosine similarity, improving recall without maintaining manual synonym lists.