Lexing

The MrKWatkins.Ast.Lexing namespace provides the plumbing that sits between a source file and a parser: a reader over the characters of a file, a token type that remembers where it came from, and a reader over the resulting tokens. It does not attempt to be a lexer generator — the rules for what makes a token are yours to write — but it takes care of the position tracking and lookahead that every hand written lexer and parser needs.

Reading Source

SourceReader is a forward-only reader over a TextFile that tracks the current Index, LineIndex and ColumnIndex as it goes. Lines are terminated by "\n", "\r\n" or a lone "\r", matching how TextFile splits its Lines.

Member	Description
Current	The character at the current position, or '\0' at the end of the file.
Peek	The character at an offset from the current position, forwards or backwards, or '\0' outside the file.
AtEnd	Whether the reader has reached the end of the file.
Advance	Moves on one character, or a given number of them, stopping at the end of the file.
AdvanceWhile	Moves on while a predicate holds, returning the number of characters consumed.

Because the end of file reads as '\0' rather than throwing, a lexer can be written without an AtEnd check on every branch.

Creating Tokens

Mark the start of a token, scan to its end, then create the token from the mark:

public enum TokenKind { Number, Identifier, Operator, EndOfFile }

private static Token<TokenKind> ReadNumber(SourceReader reader)
{
    var start = reader.Mark();
    reader.AdvanceWhile(char.IsAsciiDigit);
    return reader.CreateToken(TokenKind.Number, start);
}

Mark captures the current index, line and column as a SourceMark. CreateToken then builds a Token<TKind> running from that mark up to wherever the reader has got to.

TKind is an enum of your own describing the kinds of token in your language. Tokens are readonly record structs holding the kind, the file and the start and length of the token, so lexing a file does not allocate a token object per token. Text returns the token's characters as a ReadOnlySpan<char> over the file's text rather than as a new string.

Tokens can be constructed directly, but the constructor performs no validation for performance; going through CreateToken guarantees the index, line and column are consistent with each other.

Reading Tokens

TokenReader<TKind> hands the tokens to a parser. The last token is expected to be an end of file token, and the reader never advances past it, so Current is always safe to examine and a parser needs no bounds checks of its own. Constructing a reader with no tokens at all throws, since a lexer should always produce that end of file token.

var reader = new TokenReader<TokenKind>(tokens);

if (reader.TryConsume(TokenKind.Number, out var number))
{
    // number was consumed and the reader has advanced.
}

TryConsume advances only if the current token is of the requested kind, which is the usual shape of a recursive descent parser. Peek looks at a token an offset away, clamped to the ends of the stream, for decisions that need more than one token of lookahead.

For speculative parsing, save Position before you start and set it back afterwards to rewind:

var start = reader.Position;
if (!TryParseTypeName(reader, out var type))
{
    reader.Position = start;   // Rewind and try something else.
}

Positions from Tokens

Token<TKind>.Position converts a token to a TextFilePosition, which can be assigned straight to a node's SourcePosition. For a node built from several tokens, PositionFrom spans from a given token up to the last token consumed:

var start = reader.Current;
var expression = ParseExpression(reader);
expression.SourcePosition = reader.PositionFrom(start);

Positions for tokens at places a TextFilePosition cannot represent — a token starting on a line terminator, or a zero length token at the end of the file — are clamped to the nearest valid position rather than throwing, so an end of file token can still carry a usable position for error reporting. Empty files have no positions at all, and asking a token from one for its position throws an InvalidOperationException.

Error Recovery

SkipUntil advances until the current token is one of the kinds given, stopping at the end of file token if none is found. It always advances at least one token unless it is already at the end, which makes it safe for panic mode recovery: skipped tokens can never leave the parser looping on the same position. The synchronisation token is left as Current so the parser decides whether to consume it.

node.AddError("Expected an expression.");
reader.SkipUntil(TokenKind.Semicolon, TokenKind.CloseBrace);

Recording the failure as a message on a node rather than throwing lets the parser carry on and report every syntax error in the file in one pass.