TokenReader<TKind> Class

Definition

A reader over the Tokens lexed from a file, for consumption by a parser. The last token should be an end of file token; the reader never advances beyond it, so a parser can always safely examine Current.

public sealed class TokenReader<TKind>
   where TKind : struct, Enum

View source

Type Parameters

Name	Description
`TKind`	The type of the enum that identifies the different kinds of token.

Constructors

Name	Description
`TokenReader(IEnumerable<Token<TKind>>)`	Initialises a new instance of the `TokenReader<TKind>` class with the specified tokens.

Properties

Name	Description
`AtEnd`	Returns `true` if the reader is positioned at the last token, i.e. the end of file token, `false` otherwise.
`Count`	The number of tokens in the reader.
`Current`	The token at the current position.
`Position`	The index of the current position in the reader. Can be set to a previously retrieved value to rewind the reader, e.g. after speculative lookahead.

Methods

Name	Description
`Advance()`	Advances the reader one token. Does nothing if the reader is positioned at the last token.
`Peek(Int32)`	Returns the token at the specified offset from the current position, clamped to the ends of the reader, i.e. offsets before the first token return the first token and offsets after the last token return the last token.
`PositionFrom(Token<TKind>)`	Creates a `TextFilePosition` spanning from the specified token up to and including the last token consumed. Typically used to set the position of a node from the token that started it once parsing of the node has finished.
`SkipUntil(ReadOnlySpan<TKind>)`	Advances the reader until the current token is one of the specified kinds, stopping at the last token if none is found. Always advances at least one token unless already positioned at the last, making it suitable for panic mode error recovery: skipped tokens can never cause an infinite loop, and the synchronisation token is left as `Current` for the parser to decide whether to consume.
`TryConsume(TKind, Token<TKind>)`	If the token at the current position is of the specified kind then it is returned and the reader advanced, otherwise the stream does not move.
`TryConsume(TKind)`	If the token at the current position is of the specified kind then the reader advances, otherwise the reader does not move.