AGPascalTokenizer/DOC_EN.MD

AGPascalTokenizer

Tokenizer Library for Delphi/FPC. Based on PyPascalTokenizer work by Artem Gavrilov.

API

TAGToken struct

TAGToken is a record type of:

• Text: string with token text.
• &begin: TAGTokenizerPos with start position of token.
• &end: TAGTokenizerPos with end position of token.
• ended: boolean flag, true if it was the last token.

Has Create constructor with all values.

TAGTokenizerPos

record, saves position of token if format:

• x: integer platform-depend-base (NEXTGEN) character index in line.
• y: integer 0-based line index.

Class TAGPasTokenizer

Main class of tokenizer.

• get_next(): TAGToken function, returns. Get next token and change pos
• pos: TAGTokenizerPos property, containing position whear tokenizer was stoped after last get_next(), writeble.
• is_ended: boolean property, check if text ended.

Has constructor Create with input: TStrings param.

Class TAGPasTokenizerStack

Stack of tokens automaticaly filled from input.

• Push(t: TAGToken) push token to stack.
• Pop(): TAGToken pop token from stack.
• Last: TAGToken property, seek stask head.
• ended: boolean property, check if text ended.

Has constructor Create with input: TStrings and GetComments: boolean=True param.

Class TAGPasTokenizerParallelStack

Stack of tokens automaticaly filled from input(in second tread).

• Push(t: TAGToken) push token to stack.
• Pop(): TAGToken pop token from stack.
• Last: TAGToken property, seek stask head.
• ended: boolean property, check if text ended.

Has constructor Create with input: TStrings, GetComments: boolean=True and stackMax:integer=1000 param.

Utils

Helper functions to analyze token text.

• IsName(s: string): boolean Check for valid identifier (can be reserved word too).
• IsComment(s: string): boolean Check for valid comment.
• IsString(s: string): boolean Check for string constant.

Author

Artem Gavrilov and Vladislav Utkin.