Skip to main content
Definition Website Computer and Technology

Lexical Analysis

Subject: Computer and Technology noun /ˈlɛk.sɪ.kəl əˈnæl.ə.sɪs/ 1 definition

  1. Definition (noun)

    Entry #84 · Updated

    Lexical Analysis (also called scanning or tokenization) is the initial front-end phase of a compiler, interpreter, or text processor that reads a raw stream of source characters and groups them into a sequence of meaningful units called tokens. A dedicated component known as a lexical analyzer, lexer, or scanner matches character sequences (lexemes) against lexical rules to classify constructs—such as keywords, identifiers, literals, operators, and punctuation—while discarding insignificant whitespace and comments and recording source coordinates for error reporting. Unlike syntactic analysis (parsing), which verifies grammatical structure and builds hierarchical parse trees, lexical analysis operates strictly at the flat pattern-matching level—typically modeled with regular expressions and finite automata—without checking whether the resulting token stream forms valid statements or expressions.

    Common Forms

    • Table-Driven Lexical Analysis: Uses lexer generators such as lex and flex to compile declarative regular-expression rules into deterministic finite automata (DFAs) represented as state-transition tables, delivering predictable linear-time scanning from maintainable specifications.

    • Hand-Coded Lexical Analysis: Implements character loops, multi-character lookahead, and state transitions directly in source code; compilers such as Clang, GCC, and Go use this approach for maximum throughput, custom error recovery, and context-sensitive edge cases.

    • Incremental Lexical Analysis: Preserves per-line or per-block lexer state snapshots so code editors, syntax highlighters, and language servers can re-tokenize only modified regions around an edit instead of rescanning an entire file.

    • Streaming (On-Demand) Lexical Analysis: Emits tokens lazily one at a time when pulled by the parser—typically over a sliding input buffer—avoiding the memory cost of materializing a complete token list for large files.

    Example

    When a C compiler processes the source line int total = count + 42; // initialize, its lexical analyzer applies longest-match rules to emit seven tokens: the keyword int, identifier total, assignment operator =, identifier count, addition operator +, integer literal 42, and semicolon ;. It strips out the separating spaces and discards the // initialize comment while annotating each token with its line and column coordinates before handing the stream to the parser.