TokenReader<TKind> Class

Definition

A reader over the Tokens lexed from a file, for consumption by a parser. The last token should be an end of file token; the reader never advances beyond it, so a parser can always safely examine Current.

public sealed class TokenReader<TKind>
   where TKind : struct, Enum

View source

Type Parameters

Name	Description
TKind	The type of the enum that identifies the different kinds of token.

Constructors

Name	Description
TokenReader(IEnumerable<Token<TKind>>)	Initialises a new instance of the TokenReader<TKind> class with the specified tokens.

Properties

Name	Description
AtEnd	Returns true if the reader is positioned at the last token, that is, the end of file token, false otherwise.
Count	The number of tokens in the reader.
Current	The token at the current position.
Position	The index of the current position in the reader. Can be set to a previously retrieved value to rewind the reader, for example, after speculative lookahead.

Methods

Name	Description
Advance()	Advances the reader one token. Does nothing if the reader is positioned at the last token.
Peek(Int32)	Returns the token at the specified offset from the current position, clamped to the ends of the reader, that is, offsets before the first token return the first token and offsets after the last token return the last token.
PositionFrom(Token<TKind>)	Creates a TextFilePosition spanning from the specified token up to and including the last token consumed. Typically used to set the position of a node from the token that started it once parsing of the node has finished.
SkipUntil(ReadOnlySpan<TKind>)	Advances the reader until the current token is one of the specified kinds, stopping at the last token if none is found. Always advances at least one token unless already positioned at the last, making it suitable for panic mode error recovery: skipped tokens can never cause an infinite loop, and the synchronisation token is left as Current for the parser to decide whether to consume.
TryConsume(TKind, Token<TKind>)	If the token at the current position is of the specified kind then it is returned and the reader advanced, otherwise the stream does not move.
TryConsume(TKind)	If the token at the current position is of the specified kind then the reader advances, otherwise the reader does not move.