Class Tokenizer

code

codeLines

index

tokens

Static ReadonlyCommentChar

Static ReadonlyReserved

size

• get size(): number

• Returns the total number of tokens.

  Returns number

getLine

• getLine(index: number): { fromIndex: number; line: string }

• Retrieves the line of code corresponding to the specified index.

  Parameters

  • index: number

    The index of the character in the code.

  Returns { fromIndex: number; line: string }

  The line of code and its starting index.

locate

• locate(token?: IToken): { fromIndex: number; line: string }

• Locates the line of code containing the specified token.

  Parameters

  • Optionaltoken: IToken

    The token to locate (defaults to the current token).

  Returns { fromIndex: number; line: string }

  The line of code and its starting index.

token

• token(index?: number): null | IToken

• Retrieves a token by its index.

  Parameters

  • index: number = ...

    The index of the token (defaults to the current token index).

  Returns null | IToken

  The token at the specified index, or null if the index is out of range.

StaticReservedTokenInfo

• ReservedTokenInfo(
      token: IToken,
  ): {
      "0": boolean;
      "1": boolean;
      copy: boolean;
      delete: boolean;
      fileHead: boolean;
      log: boolean;
      pointerNext: boolean;
      pointerPrevious: boolean;
      print: boolean;
  }

• Determines if a given token matches a reserved token and returns detailed token information.

  Parameters

  • token: IToken

    The token to check.

  Returns {
      "0": boolean;
      "1": boolean;
      copy: boolean;
      delete: boolean;
      fileHead: boolean;
      log: boolean;
      pointerNext: boolean;
      pointerPrevious: boolean;
      print: boolean;
  }

  An object mapping reserved token types to boolean values indicating a match.

StaticTokenize

• Tokenize(code: string): Tokenizer

• Tokenizes a code string into an array of IToken objects.

  Parameters

  • code: string

    The code string to tokenize.

  Returns Tokenizer

  A new instance of the Tokenizer class containing the tokens and the original code.