Class std.re.Lexer

Synopsis

public abstract class Lexer

Lexer (lexical analyzer).

This is an abstract class, to which you can make subclasses to parse other languages. In each subclass, there would be a constructor which calls addTokenType() to define the types of tokens in the language it is parsing. Then at some point later, or possibly in the constructor itself, you would call feed() to provide data to be parsed. Then, for each token spotted, onToken() will be called, and your subclass decides what to do with it.

Fields

Constructors

Methods

protected final Object addTokenType(String regex,bool whitespace);

Add a new token type.

Arguments

regex	The regular expression identifying this token type.
whitespace	If `true`, then this is a whitespace token and should be ignored and not passed to `onToken()`.

Returns

An object which will later be passed onToken() when identifying this token type.

protected final Object addTokenType(String regex);

Add a new non-whitespace token type.

Arguments

regex	The regular expression identifying this token type.

Returns

An object which will later be passed onToken() when identifying this token type.

public final Object addWhitespaceTokenType(String regex);

Add a whitespace token type.

Arguments

regex	The regular expression identifying this token type.

Returns

An object which will later be passed onToken() when identifying this token type.

protected abstract void onToken(Object type,int lineno,Match match);

This function is called every time a non-whitespace token is encountered by the parser.

Arguments

type	The token type (previously returned by `addTokenType()`).
lineno	The line number within the `feed()` data.
match	The `std.re.Match` object representing the matched token.

protected final void feed(String data);

Feed the parser.

Arguments

data	The data to be parsed.