Interface Tokenizer<T>

  • Type Parameters:
    T - given type
    All Known Implementing Classes:
    RegexTokenizer

    interface Tokenizer<T>
    A tokenizer. Can produce arrays of tokens from a given type.
    Since:
    1.0
    • Method Detail

      • tokenize

        T[] tokenize​(java.lang.CharSequence text)
        Returns an array of tokens.
        Parameters:
        text - input text
        Returns:
        array of tokens