artem3213212/AGPascalTokenizer
1
0
Fork 0
Code Issues 3 Pull Requests Releases Wiki Activity

Add Multi threading

Browse Source
This commit is contained in:
Artem3213212
2019-04-15 19:27:49 +03:00
parent 67bb7b5fed
commit 94d7217bbe
5 changed files with 619 additions and 666 deletions
Download Patch File Download Diff File Expand all files Collapse all files

121
DOC_EN.MD
View File

@@ -1,54 +1,67 @@
# AGPascalTokenizer
Tokenizer Library for Delphi/FPC. Based on [PyPascalTokenizer](https://github.com/Artem3213212/PyPascalTokenizer) work by [Artem Gavrilov](https://github.com/Artem3213212).
## API
### TToken struct
TToken is a `record` type of:
* Text: `string` with token text.
* &begin: `TTokenizerPos` with start position of token.
* &end: `TTokenizerPos` with end position of token.
* ended: `boolean` flag, true if it was the last token.
Has Create constructor with all values.
### TTokenizerPos
`record`, saves position of token if format:
* x: `integer` platform-depend-base (`NEXTGEN`) character index in line.
* y: `integer` 0-based line index.
### `Class` PasTokenizer
Main class of tokenizer.
* s: `TStrings` private, use constructor. List of source code strings.
* get_next(): function, returns `TToken`. Get next token and change end pos
* read_next(): function, not implemented, get next token, but don't change end pos.
* is_ended(): function, not implemented, check if text ended.
Has constructor Create with `s` param.
### `Class` PasTokenizerStack
Not implemented.
### `Class` PasTokenizerParallelStack
Not implemented.
## Utils
Helper functions to analyze token text.
* is_name(s: `string`): `boolean` Check for valid identifier (can be reserved word too).
* is_comment(s: `string`): `boolean` Check for valid comment.
* is_string(s: `string`): `boolean` Check for string constant.
## Author
[Artem Gavrilov](https://github.com/Artem3213212) and [Vladislav Utkin](https://github.com/viad00).
# AGPascalTokenizer
Tokenizer Library for Delphi/FPC. Based on [PyPascalTokenizer](https://github.com/Artem3213212/PyPascalTokenizer) work by [Artem Gavrilov](https://github.com/Artem3213212).
## API
### TAGToken struct
TAGToken is a `record` type of:
* Text: `string` with token text.
* &begin: `TAGTokenizerPos` with start position of token.
* &end: `TAGTokenizerPos` with end position of token.
* ended: `boolean` flag, true if it was the last token.
Has Create constructor with all values.
### TAGTokenizerPos
`record`, saves position of token if format:
* x: `integer` platform-depend-base (`NEXTGEN`) character index in line.
* y: `integer` 0-based line index.
### `Class` TAGPasTokenizer
Main class of tokenizer.
* get_next(): `TAGToken` function, returns. Get next token and change pos
* pos: `TAGTokenizerPos` property, containing position whear tokenizer was stoped after last get_next(), writeble.
* is_ended: `boolean` property, check if text ended.
Has constructor Create with `input`: `TStrings` param.
### `Class` TAGPasTokenizerStack
Stack of tokens automaticaly filled from input.
* Push(t: `TAGToken`) push token to stack.
* Pop(): `TAGToken` pop token from stack.
* Last: `TAGToken` property, seek stask head.
* ended: `boolean` property, check if text ended.
Has constructor Create with input: `TStrings` and GetComments: `boolean=True` param.
### `Class` TAGPasTokenizerParallelStack
Stack of tokens automaticaly filled from input(in second tread).
* Push(t: `TAGToken`) push token to stack.
* Pop(): `TAGToken` pop token from stack.
* Last: `TAGToken` property, seek stask head.
* ended: `boolean` property, check if text ended.
Has constructor Create with input: `TStrings`, GetComments: `boolean=True` and stackMax:`integer=1000` param.
## Utils
Helper functions to analyze token text.
* IsName(s: `string`): `boolean` Check for valid identifier (can be reserved word too).
* IsComment(s: `string`): `boolean` Check for valid comment.
* IsString(s: `string`): `boolean` Check for string constant.
## Author
[Artem Gavrilov](https://github.com/Artem3213212) and [Vladislav Utkin](https://github.com/viad00).