lexer.l0

lexer.l0

Module: lexer

Source: compiler/stage2_l0/src/lexer.l0 Language: Dea/L0

Imports / Includes

  • std.linear_map
  • std.vector
  • util.strings
  • std.text
  • std.unit
  • tokens
  • std.string
  • util.diag
  • std.io

Symbols

Function is_ident_start

1

func is_ident_start(c: byte) -> bool

Check whether a byte can start an identifier.

Parameters:

  • c: Byte to inspect.

Returns: true if c is alphabetic or _.

Function is_ident_part

1

func is_ident_part(c: byte) -> bool

Check whether a byte can continue an identifier.

Parameters:

  • c: Byte to inspect.

Returns: true if c is alphanumeric or _.

Function is_escape_char

1

func is_escape_char(c: byte) -> bool

Check whether a byte is a supported escape-code character.

Parameters:

  • c: Byte to inspect.

Returns: true if c is accepted after a backslash in literals.

Function escape_char_value

1

func escape_char_value(c: byte) -> byte

Decode one simple escape-code character into its byte value.

Parameters:

  • c: Escape-code byte after the backslash.

Returns: Decoded byte value.

Function is_octal_digit

1

func is_octal_digit(c: byte) -> bool

Check whether a byte is an octal digit.

Parameters:

  • c: Byte to inspect.

Returns: true if c is in the range 0..7.

Function is_hex_digit

1

func is_hex_digit(c: byte) -> bool

Check whether a byte is a hexadecimal digit.

Parameters:

  • c: Byte to inspect.

Returns: true if c is in 0..9, a..f, or A..F.

Function hex_char_to_int

1

func hex_char_to_int(c: byte) -> int

Convert a hexadecimal digit character to its integer value.

Parameters:

  • c: Hexadecimal digit byte.

Returns: Integer value in the range 0..15, or -1 if c is not valid.

Function is_printable_ascii

1

func is_printable_ascii(c: byte) -> bool

Check whether a byte is a printable ASCII character (space through tilde).

Parameters:

  • c: Byte to inspect.

Returns: true if c is a space, tab, carriage return, or newline.

Function ls_create

1

func ls_create(source: string, filename: string) -> LexerState*

Create a lexer state for source text and filename metadata.

Parameters:

  • source: Source text to tokenize.
  • filename: Source filename used in diagnostics.

Returns: New lexer state.

Function ls_has_errors

1

func ls_has_errors(self: LexerState*) -> bool

Report whether lexing has produced any errors.

Parameters:

  • self: Lexer state to inspect.

Returns: true if the lexer’s diagnostic collector contains an error.

Function ls_emit_error

1

func ls_emit_error(self: LexerState*, code: string, message: string, line: int, column: int, line_end: int, column_end: int)

Emit a lexer diagnostic at an explicit source span.

Parameters:

  • self: Lexer state collecting diagnostics.
  • code: Diagnostic code.
  • message: Diagnostic message.
  • line: One-based starting line.
  • column: One-based starting column.
  • line_end: One-based ending line.
  • column_end: One-based ending column.

Function ls_queue_token

1

func ls_queue_token(self: LexerState*, token: Token)

Queue a physical token for later emission by ls_next_token .

Parameters:

  • self: Lexer state collecting queued tokens.
  • token: Token to queue.

Function ls_has_queued_tokens

1

func ls_has_queued_tokens(self: LexerState*) -> bool

Return whether previously queued tokens remain to be emitted.

Parameters:

  • self: Lexer state to inspect.

Returns: true when queued tokens remain.

Function ls_take_queued_token

1

func ls_take_queued_token(self: LexerState*) -> Token

Return the next queued token and transfer ownership out of the queue.

Parameters:

  • self: Lexer state to inspect.

Returns: Next queued token.

Function ls_defer_recoverable_error

1

func ls_defer_recoverable_error(self: LexerState*, code: string, message: string, line: int, column: int, line_end: int, column_end: int)

Store a recoverable lexer diagnostic for the token currently being scanned.

Function ls_clear_pending_error

1

func ls_clear_pending_error(self: LexerState*)

Clear pending recoverable diagnostic state before scanning a token.

Function ls_queue_pending_errors

1

func ls_queue_pending_errors(self: LexerState*, start_index: int)

Queue all pending recoverable diagnostics as unrecoverable lexer-error tokens.

Parameters:

  • self: Lexer state to drain.
  • start_index: Physical token start offset.

Function ls_queue_pending_recovery

1

func ls_queue_pending_recovery(self: LexerState*, start_index: int, start_line: int, start_column: int, recovery: TokenRecovery) -> Token

Queue pending recoverable diagnostics and a final recoverable wrapper token.

Parameters:

  • self: Lexer state to drain.
  • start_index: Physical token start offset.
  • start_line: Physical token start line.
  • start_column: Physical token start column.
  • recovery: Logical recovery payload for the final wrapper.

Returns: First queued token.

Function ls_queue_terminal_recovery

1

func ls_queue_terminal_recovery(self: LexerState*, start_index: int, start_line: int, start_column: int, code: string, message: string, line_end: int, column_end: int, recovery: TokenRecovery) -> Token

Queue pending recoverable diagnostics followed by a terminal recoverable wrapper token.

Parameters:

  • self: Lexer state to drain.
  • start_index: Physical token start offset.
  • start_line: Physical token start line.
  • start_column: Physical token start column.
  • code: Terminal diagnostic code.
  • message: Terminal diagnostic message.
  • line_end: Terminal diagnostic end line.
  • column_end: Terminal diagnostic end column.
  • recovery: Logical recovery payload for the final wrapper.

Returns: First queued token.

Function ls_free

1

func ls_free(self: LexerState*)

Free a lexer state and its diagnostics.

Parameters:

  • self: Lexer state to free.

Function ls_at_end

1

func ls_at_end(self: LexerState*) -> bool

Check whether the lexer cursor is at the end of the source.

Parameters:

  • self: Lexer state to inspect.

Returns: true if no more source bytes remain.

Function ls_peek

1

func ls_peek(self: LexerState*) -> byte

Return the current source byte without advancing.

Parameters:

  • self: Lexer state to inspect.

Returns: Current source byte, or \\0 at end of input.

Function ls_peek_next

1

func ls_peek_next(self: LexerState*) -> byte

Return the byte after the current source byte without advancing.

Parameters:

  • self: Lexer state to inspect.

Returns: Next source byte, or \\0 if there is no next byte.

Function ls_advance

1

func ls_advance(self: LexerState*) -> byte

Consume and return the current source byte.

The column advances once per Unicode code point, not per byte: UTF-8 continuation bytes do not bump it, so positions stay aligned with the code-point columns reported by the Stage 1 Python frontend.

Parameters:

  • self: Lexer state to advance.

Returns: Byte that was current before advancing.

Function tokenize

1

func tokenize(self: LexerState*) -> TokenVector?

Tokenizes the input source code and returns a vector of tokens.

If a lexing error occurs, the error field in the LexerState will be set and null will be returned.

Function ls_next_token

1

func ls_next_token(self: LexerState*) -> Token?

Reads the next token from the input source code.

Returns a Token if successful, or null if a lexing error occurs, in which case the error field in the LexerState will be set.

Function ls_read_byte_literal

1

func ls_read_byte_literal(self: LexerState*, start_index: int, start_line: int, start_column: int) -> Token?

Reads a byte literal from the input, starting after the opening single quote.

Handles escape sequences and validates that the literal represents a single byte. Returns a Token of type TT_BYTE if successful, or null if a lexing error occurs, in which case the error field in the LexerState will be set.

Function ls_read_string_literal

1

func ls_read_string_literal(self: LexerState*, start_index: int, start_line: int, start_column: int) -> Token?

Reads a string literal from the input, starting after the opening double quote.

Handles escape sequences and validates that the string is properly terminated. Returns a Token of type TT_STRING if successful, or null if a lexing error occurs, in which case the error field in the LexerState will be set.

Function ls_read_valid_char_escape

1

func ls_read_valid_char_escape(self: LexerState*, single_byte: bool) -> EscapedChar?

Reads a valid escape sequence from the input, starting after the backslash.

Validates that the escape sequence is well-formed and returns both the original escape string and its corresponding character value. If single_byte is true, also validates that the resulting character value can fit in a single byte (0-255). Returns an EscapedChar struct if successful, or null if an invalid escape sequence is encountered, in which case the error field in the LexerState will be set.

Function ls_read_number

1

func ls_read_number(self: LexerState*, first_char: byte, is_negative: bool, start_index: int, start_line: int, start_column: int) -> Token?

Reads an integer literal from the input, starting with the first digit character.

Validates that the literal is well-formed and does not contain invalid characters immediately following it. Also checks for integer overflow and ensures the value fits within a 32-bit signed integer range. Returns a Token of type TT_INT if successful, or null if a lexing error occurs, in which case the error field in the LexerState will be set.

Function ls_skip_whitespace_and_comments

1

func ls_skip_whitespace_and_comments(self: LexerState*) -> Unit?

Skips over any whitespace characters and comments in the input until it reaches a non-whitespace, non-comment character or the end of the input.

Handles both line comments and block comments. Returns std.unit::present() if successful, or null if an unterminated block comment is encountered, in which case the error field in the LexerState will be set.

Function ls_skip_invalid_characters

1

func ls_skip_invalid_characters(self: LexerState*)

Skips over the rest of an invalid-character run after the first invalid byte has been consumed.

The run stops at printable ASCII or whitespace, so the resulting diagnostic span never crosses a line break.

Struct LexerState

Lexer state object that keeps track of the current position in the source code being lexed.

LexerState Field source

1

source: string

LexerState Field filename

1

filename: string

LexerState Field length

1

length: int

LexerState Field index

1

index: int

LexerState Field line

1

line: int

LexerState Field column

1

column: int

LexerState Field prev_ends_expression

1

prev_ends_expression: bool

LexerState Field diags

1

diags: DiagCollector*

LexerState Field pending_errors

1

pending_errors: DiagnosticVector*

LexerState Field queued_tokens

1

queued_tokens: TokenVector

LexerState Field queued_index

1

queued_index: int

Struct EscapedChar

Represents the EscapedChar structure.

EscapedChar Field escape_str

1

escape_str: string

EscapedChar Field char_value

1

char_value: int