Interface Tokenizer<T>
- Type Parameters:
T
- given type
- All Known Implementing Classes:
RegexTokenizer
interface Tokenizer<T>
A tokenizer. Can produce arrays of tokens from a given type.
- Since:
- 1.0
-
Method Summary
-
Method Details
-
tokenize
Returns an array of tokens.- Parameters:
text
- input text- Returns:
- array of tokens
-