Module: tokens

Source: compiler/stage2_l0/src/tokens.l0 Language: Dea/L0

Imports / Includes

• std.vector
• std.text
• sys.rt
• std.linear_map
• std.unit
• std.string
• std.io

Symbols

• TokenRecovery
• TokenType
• token_to_string
• token_recovery_to_string
• token_recovery_is_some
• token_recovered
• token_logical_predecessor_ends_expression
• is_reserved_ident_word
• is_reserved_keyword
• ident_to_token
• is_expr_ending_token_type
• tv_create
• tv_push
• tv_get
• tv_len
• token_release_payload
• token_recovery_release_payload
• tv_free
• Token
• TokenVector

Enum TokenRecovery

TokenRecovery

Non-recursive logical token payload used by lexer-error wrappers.

Variants:

• TR_NONE
• TR_INT(text: string, value: int)
• TR_BYTE(text: string, value: byte)
• TR_STRING(text: string, value: string)

Enum TokenType

TokenType

Defines the TokenType enumeration.

Variants:

• TT_EOF
• TT_IDENT(text: string)
• TT_UNDERSCORE
• TT_INT(text: string, value: int)
• TT_BYTE(text: string, value: byte)
• TT_STRING(text: string, value: string)
• TT_MODULE
• TT_IMPORT
• TT_FUNC
• TT_STRUCT
• TT_ENUM
• TT_TYPE
• TT_EXTERN
• TT_LET
• TT_RETURN
• TT_MATCH
• TT_CASE
• TT_IF
• TT_ELSE
• TT_WHILE
• TT_FOR
• TT_BREAK
• TT_CONTINUE
• TT_TRUE
• TT_FALSE
• TT_NULL
• TT_AS
• TT_NEW
• TT_DROP
• TT_WITH
• TT_CLEANUP
• TT_LBRACE
• TT_RBRACE
• TT_LPAREN
• TT_RPAREN
• TT_LBRACKET
• TT_RBRACKET
• TT_COMMA
• TT_SEMI
• TT_COLON
• TT_DOUBLE_COLON
• TT_ARROW_FUNC
• TT_ARROW_MATCH
• TT_EQ
• TT_PLUS
• TT_MINUS
• TT_STAR
• TT_SLASH
• TT_MODULO
• TT_LT
• TT_GT
• TT_LE
• TT_GE
• TT_EQEQ
• TT_NE
• TT_ANDAND
• TT_OROR
• TT_BANG
• TT_QUESTION
• TT_DOT
• TT_AMP
• TT_PIPE
• TT_CARET
• TT_TILDE
• TT_LSHIFT
• TT_RSHIFT
• TT_FUTURE_EXTENSION
• TT_LEXER_ERROR(code: string, message: string, line_end: int, column_end: int, recovery: TokenRecovery)

Function token_to_string

func token_to_string(token: Token) -> string

Render a token as human-readable text.

Parameters:

• token: Token to format.

Returns: Readable token spelling for diagnostics and debugging output.

Function token_recovery_to_string

func token_recovery_to_string(recovery: TokenRecovery) -> string

Render a logical recovery payload.

Parameters:

• recovery: Recovery payload to format.

Returns: Human-readable recovery token spelling.

Function token_recovery_is_some

func token_recovery_is_some(recovery: TokenRecovery) -> bool

Return whether a lexer-error wrapper has a parser-visible recovery token.

Parameters:

• recovery: Recovery payload to inspect.

Returns: true if the payload represents a logical token.

Function token_recovered

func token_recovered(wrapper: Token) -> Token

Build the logical token represented by a lexer-error recovery payload.

Parameters:

• wrapper: Physical lexer-error token.

Returns: Logical recovered token, or the wrapper when there is no recovery.

Function token_logical_predecessor_ends_expression

func token_logical_predecessor_ends_expression(token: Token, previous: bool) -> bool

Return whether the parser-visible predecessor ends an expression.

Lexer-error wrappers with recovery contribute the recovered token context; wrappers without recovery preserve the preceding logical context.

Parameters:

• token: Newly scanned physical token.
• previous: Whether the previous parser-visible token ended an expression.

Returns: Expression-ending context for signed-literal lexing.

Function is_reserved_ident_word

func is_reserved_ident_word(word: string) -> bool

Check whether identifier text is reserved as a builtin type name.

Parameters:

• word: Identifier text to inspect.

Returns: true if word is reserved and cannot be used as a plain identifier.

Function is_reserved_keyword

func is_reserved_keyword(token: Token) -> bool

Check whether a token is reserved as a keyword or future extension.

Parameters:

• token: Token to inspect.

Returns: true if the token cannot be used as a variable name.

Function ident_to_token

func ident_to_token(ident: string, index: int, line: int, col: int) -> Token

Convert identifier text into the appropriate token at a source location.

Parameters:

• ident: Identifier text as it appeared in source.
• index: Zero-based byte offset in the source.
• line: One-based source line.
• col: One-based source column.

Returns: Keyword, underscore, future-extension, or identifier token for ident.

Function is_expr_ending_token_type

func is_expr_ending_token_type(token_type: TokenType) -> bool

Check whether a token type can legally terminate an expression.

Parameters:

• token_type: Token type to inspect.

Returns: true if the token type can appear at the end of an expression.

Function tv_create

func tv_create() -> TokenVector

Create an empty token vector.

Returns: Token vector with an initial capacity.

Function tv_push

func tv_push(self: TokenVector, token: Token)

Append a token to a token vector.

Parameters:

• self: Token vector to extend.
• token: Token value to append.

Function tv_get

func tv_get(self: TokenVector, index: int) -> Token

Return a token from a token vector.

Parameters:

• self: Token vector to inspect.
• index: Zero-based token index.

Returns: Token at index.

Function tv_len

func tv_len(self: TokenVector) -> int

Return the number of tokens in a token vector.

Parameters:

• self: Token vector to inspect.

Returns: Token count.

Function token_release_payload

func token_release_payload(token: Token)

Release ARC-managed payload strings held by a token.

Parameters:

• token: Token whose payload should be released.

Function token_recovery_release_payload

func token_recovery_release_payload(recovery: TokenRecovery)

Release ARC-managed payload strings held by a token recovery payload.

Parameters:

• recovery: Recovery payload whose strings should be released.

Function tv_free

func tv_free(self: TokenVector)

Free a token vector and release any token payload strings.

Parameters:

• self: Token vector to free.

Struct Token

Represents the Token structure.

Token Field token_type

token_type: TokenType

Token Field index

index: int

Token Field line

line: int

Token Field column

column: int

Struct TokenVector

Represents the TokenVector structure.

TokenVector Field vec

vec: VectorBase*