Method Search.Grammar.Lexer.tokenize()
- Method tokenize
string|array(array(Token|string)) tokenize(string query)
- Description
Tokenizes a query into tokens for later use by a parser.
- Parameter query
The query to tokenize.
- Returns
An array containing the tokens: ({ ({ TOKEN_WORD, "foo" }), ... }) Or, in case of an error, a string with the error message.