Lexical Analysis
Subject: Computer and Technology noun /ˈlɛk.sɪ.kəl əˈnæl.ə.sɪs/ 1 definition
- compiler-design
- programming-languages
- formal-languages
- computer-science
-
Definition (noun)
Entry #84 · UpdatedLexical Analysis (also called scanning or tokenization) is the initial front-end phase of a compiler, interpreter, or text processor that reads a raw stream of source characters and groups them into a sequence of meaningful units called tokens. A dedicated component known as a lexical analyzer, lexer, or scanner matches character sequences (lexemes) against lexical rules to classify constructs—such as keywords, identifiers, literals, operators, and punctuation—while discarding insignificant whitespace and comments and recording source coordinates for error reporting. Unlike syntactic analysis (parsing), which verifies grammatical structure and builds hierarchical parse trees, lexical analysis operates strictly at the flat pattern-matching level—typically modeled with regular expressions and finite automata—without checking whether the resulting token stream forms valid statements or expressions.
Common Forms
Table-Driven Lexical Analysis: Uses lexer generators such as
lexandflexto compile declarative regular-expression rules into deterministic finite automata (DFAs) represented as state-transition tables, delivering predictable linear-time scanning from maintainable specifications.Hand-Coded Lexical Analysis: Implements character loops, multi-character lookahead, and state transitions directly in source code; compilers such as Clang, GCC, and Go use this approach for maximum throughput, custom error recovery, and context-sensitive edge cases.
Incremental Lexical Analysis: Preserves per-line or per-block lexer state snapshots so code editors, syntax highlighters, and language servers can re-tokenize only modified regions around an edit instead of rescanning an entire file.
Streaming (On-Demand) Lexical Analysis: Emits tokens lazily one at a time when pulled by the parser—typically over a sliding input buffer—avoiding the memory cost of materializing a complete token list for large files.
Example
When a C compiler processes the source line
int total = count + 42; // initialize, its lexical analyzer applies longest-match rules to emit seven tokens: the keywordint, identifiertotal, assignment operator=, identifiercount, addition operator+, integer literal42, and semicolon;. It strips out the separating spaces and discards the// initializecomment while annotating each token with its line and column coordinates before handing the stream to the parser.