Lexical analyzer and C++ text scanner

This lexical analyzer use DFAs (Deterministic Finite Automata) to match tokens found on the character input stream. DFAs are defined by the set of regular expressions and are created at runtime. More than one automaton (lexical class, lexical tree) may be defined.

Actions attached to regular expressions (which are executed when that expression has been matched on the input stream) may switch from one lexical class to another.

The C++ text scanner is the example of using the lexical analyzer.

The LEXER unit contains:

• TTokenInfo is a structure, which describes a regular expression and contains a pointer to the action function;
• TLexNode is the class of the DFA tree node;
• TNfaNode is the class of the NFA tree node;
• TNfaRoot is the helper class of the root node of the NFA tree;
• TLexTree is a class that contains the array of root nodes of the DFA trees and encapsulates the conversion of the set of regular expressions to DFAs by using NFAs;
• TToken is the class of output tokens;
• TErrorList is the class of the string list of errors and warnings;
• TLexer is the base class of the lexical analyzer.

The CPLEXER unit contains:

• the description set of regular expressions and action functions for the C++ language with the Borland extensions;
• TPreScan is a helper class, which deletes from the input stream the EOF symbols and the backslash with the following newline symbols;
• TCPPScan is the C++ scanner class.

The SCAN unit contains the MAIN function of the C++ scanner sample.

Nikolay Antonov & Vyatcheslav Baranov. MBLab, 5/14/2002.

You can download the source code from these link.

    (29 kB)

Lexical analyzer and C++ text scanner     Updated 08/24/02