Class StringTokenizer

  • All Implemented Interfaces:
    java.lang.Cloneable, java.util.Iterator<java.lang.String>, java.util.ListIterator<java.lang.String>
    Direct Known Subclasses:
    TextStringBuilder.TextStringBuilderTokenizer

    public class StringTokenizer
    extends java.lang.Object
    implements java.util.ListIterator<java.lang.String>, java.lang.Cloneable
    Tokenizes a string based on delimiters (separators) and supporting quoting and ignored character concepts.

    This class can split a String into many smaller strings. It aims to do a similar job to StringTokenizer, however it offers much more control and flexibility including implementing the ListIterator interface. By default, it is set up like StringTokenizer.

    The input String is split into a number of tokens. Each token is separated from the next String by a delimiter. One or more delimiter characters must be specified.

    Each token may be surrounded by quotes. The quote matcher specifies the quote character(s). A quote may be escaped within a quoted section by duplicating itself.

    Between each token and the delimiter are potentially characters that need trimming. The trimmer matcher specifies these characters. One usage might be to trim whitespace characters.

    At any point outside the quotes there might potentially be invalid characters. The ignored matcher specifies these characters to be removed. One usage might be to remove new line characters.

    Empty tokens may be removed or returned as null.

     "a,b,c"         - Three tokens "a","b","c"   (comma delimiter)
     " a, b , c "    - Three tokens "a","b","c"   (default CSV processing trims whitespace)
     "a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched)
     
    StringTokenizer properties and options
    Property Type Default
    delim CharSetMatcher { \t\n\r\f}
    quote NoneMatcher {}
    ignore NoneMatcher {}
    emptyTokenAsNull boolean false
    ignoreEmptyTokens boolean true
    Since:
    1.3
    • Constructor Summary

      Constructors 
      Constructor Description
      StringTokenizer()
      Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer, but with no text to tokenize.
      StringTokenizer​(char[] input)
      Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer.
      StringTokenizer​(char[] input, char delim)
      Constructs a tokenizer splitting on the specified character.
      StringTokenizer​(char[] input, char delim, char quote)
      Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
      StringTokenizer​(char[] input, java.lang.String delim)
      Constructs a tokenizer splitting on the specified string.
      StringTokenizer​(char[] input, StringMatcher delim)
      Constructs a tokenizer splitting using the specified delimiter matcher.
      StringTokenizer​(char[] input, StringMatcher delim, StringMatcher quote)
      Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
      StringTokenizer​(java.lang.String input)
      Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer.
      StringTokenizer​(java.lang.String input, char delim)
      Constructs a tokenizer splitting on the specified delimiter character.
      StringTokenizer​(java.lang.String input, char delim, char quote)
      Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
      StringTokenizer​(java.lang.String input, java.lang.String delim)
      Constructs a tokenizer splitting on the specified delimiter string.
      StringTokenizer​(java.lang.String input, StringMatcher delim)
      Constructs a tokenizer splitting using the specified delimiter matcher.
      StringTokenizer​(java.lang.String input, StringMatcher delim, StringMatcher quote)
      Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void add​(java.lang.String obj)
      Unsupported ListIterator operation.
      private void addToken​(java.util.List<java.lang.String> list, java.lang.String tok)
      Adds a token to a list, paying attention to the parameters we've set.
      private void checkTokenized()
      Checks if tokenization has been done, and if not then do it.
      java.lang.Object clone()
      Creates a new instance of this Tokenizer.
      (package private) java.lang.Object cloneReset()
      Creates a new instance of this Tokenizer.
      java.lang.String getContent()
      Gets the String content that the tokenizer is parsing.
      private static StringTokenizer getCSVClone()
      Returns a clone of CSV_TOKENIZER_PROTOTYPE.
      static StringTokenizer getCSVInstance()
      Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
      static StringTokenizer getCSVInstance​(char[] input)
      Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
      static StringTokenizer getCSVInstance​(java.lang.String input)
      Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
      StringMatcher getDelimiterMatcher()
      Gets the field delimiter matcher.
      StringMatcher getIgnoredMatcher()
      Gets the ignored character matcher.
      StringMatcher getQuoteMatcher()
      Gets the quote matcher currently in use.
      java.lang.String[] getTokenArray()
      Gets a copy of the full token list as an independent modifiable array.
      java.util.List<java.lang.String> getTokenList()
      Gets a copy of the full token list as an independent modifiable list.
      StringMatcher getTrimmerMatcher()
      Gets the trimmer character matcher.
      private static StringTokenizer getTSVClone()
      Returns a clone of TSV_TOKENIZER_PROTOTYPE.
      static StringTokenizer getTSVInstance()
      Gets a new tokenizer instance which parses Tab Separated Value strings.
      static StringTokenizer getTSVInstance​(char[] input)
      Gets a new tokenizer instance which parses Tab Separated Value strings.
      static StringTokenizer getTSVInstance​(java.lang.String input)
      Gets a new tokenizer instance which parses Tab Separated Value strings.
      boolean hasNext()
      Tests whether there are any more tokens.
      boolean hasPrevious()
      Tests whether there are any previous tokens that can be iterated to.
      boolean isEmptyTokenAsNull()
      Tests whether the tokenizer currently returns empty tokens as null.
      boolean isIgnoreEmptyTokens()
      Tests whether the tokenizer currently ignores empty tokens.
      private boolean isQuote​(char[] srcChars, int pos, int len, int quoteStart, int quoteLen)
      Tests if the characters at the index specified match the quote already matched in readNextToken().
      java.lang.String next()
      Gets the next token.
      int nextIndex()
      Gets the index of the next token to return.
      java.lang.String nextToken()
      Gets the next token from the String.
      java.lang.String previous()
      Gets the token previous to the last returned token.
      int previousIndex()
      Gets the index of the previous token.
      java.lang.String previousToken()
      Gets the previous token from the String.
      private int readNextToken​(char[] srcChars, int start, int len, TextStringBuilder workArea, java.util.List<java.lang.String> tokenList)
      Reads character by character through the String to get the next token.
      private int readWithQuotes​(char[] srcChars, int start, int len, TextStringBuilder workArea, java.util.List<java.lang.String> tokenList, int quoteStart, int quoteLen)
      Reads a possibly quoted string token.
      void remove()
      Throws UnsupportedOperationException for this unsupported ListIterator operation.
      StringTokenizer reset()
      Resets this tokenizer, forgetting all parsing and iteration already completed.
      StringTokenizer reset​(char[] input)
      Resets this tokenizer, giving it a new input string to parse.
      StringTokenizer reset​(java.lang.String input)
      Resets this tokenizer, giving it a new input string to parse.
      void set​(java.lang.String obj)
      Throws UnsupportedOperationException for this unsupported ListIterator operation.
      StringTokenizer setDelimiterChar​(char delim)
      Sets the field delimiter character.
      StringTokenizer setDelimiterMatcher​(StringMatcher delim)
      Sets the field delimiter matcher.
      StringTokenizer setDelimiterString​(java.lang.String delim)
      Sets the field delimiter string.
      StringTokenizer setEmptyTokenAsNull​(boolean emptyAsNull)
      Sets whether the tokenizer should return empty tokens as null.
      StringTokenizer setIgnoredChar​(char ignored)
      Sets the character to ignore.
      StringTokenizer setIgnoredMatcher​(StringMatcher ignored)
      Sets the matcher for characters to ignore.
      StringTokenizer setIgnoreEmptyTokens​(boolean ignoreEmptyTokens)
      Sets whether the tokenizer should ignore and not return empty tokens.
      StringTokenizer setQuoteChar​(char quote)
      Sets the quote character to use.
      StringTokenizer setQuoteMatcher​(StringMatcher quote)
      Set the quote matcher to use.
      StringTokenizer setTrimmerMatcher​(StringMatcher trimmer)
      Sets the matcher for characters to trim.
      int size()
      Gets the number of tokens found in the String.
      protected java.util.List<java.lang.String> tokenize​(char[] srcChars, int offset, int count)
      Internal method to performs the tokenization.
      java.lang.String toString()
      Gets the String content that the tokenizer is parsing.
      • Methods inherited from class java.lang.Object

        equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
      • Methods inherited from interface java.util.Iterator

        forEachRemaining
    • Field Detail

      • CSV_TOKENIZER_PROTOTYPE

        private static final StringTokenizer CSV_TOKENIZER_PROTOTYPE
        Comma separated values tokenizer internal variable.
      • TSV_TOKENIZER_PROTOTYPE

        private static final StringTokenizer TSV_TOKENIZER_PROTOTYPE
        Tab separated values tokenizer internal variable.
      • chars

        private char[] chars
        The text to work on.
      • tokens

        private java.lang.String[] tokens
        The parsed tokens.
      • tokenPos

        private int tokenPos
        The current iteration position.
      • delimMatcher

        private StringMatcher delimMatcher
        The delimiter matcher.
      • quoteMatcher

        private StringMatcher quoteMatcher
        The quote matcher.
      • ignoredMatcher

        private StringMatcher ignoredMatcher
        The ignored matcher.
      • trimmerMatcher

        private StringMatcher trimmerMatcher
        The trimmer matcher.
      • emptyAsNull

        private boolean emptyAsNull
        Whether to return empty tokens as null.
      • ignoreEmptyTokens

        private boolean ignoreEmptyTokens
        Whether to ignore empty tokens.
    • Constructor Detail

      • StringTokenizer

        public StringTokenizer()
        Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer, but with no text to tokenize.

        This constructor is normally used with reset(String).

      • StringTokenizer

        public StringTokenizer​(char[] input)
        Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer.
        Parameters:
        input - the string which is to be parsed, not cloned
      • StringTokenizer

        public StringTokenizer​(char[] input,
                               char delim)
        Constructs a tokenizer splitting on the specified character.
        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter character
      • StringTokenizer

        public StringTokenizer​(char[] input,
                               char delim,
                               char quote)
        Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter character
        quote - the field quoted string character
      • StringTokenizer

        public StringTokenizer​(char[] input,
                               java.lang.String delim)
        Constructs a tokenizer splitting on the specified string.
        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter string
      • StringTokenizer

        public StringTokenizer​(char[] input,
                               StringMatcher delim)
        Constructs a tokenizer splitting using the specified delimiter matcher.
        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter matcher
      • StringTokenizer

        public StringTokenizer​(char[] input,
                               StringMatcher delim,
                               StringMatcher quote)
        Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter character
        quote - the field quoted string character
      • StringTokenizer

        public StringTokenizer​(java.lang.String input)
        Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer.
        Parameters:
        input - the string which is to be parsed
      • StringTokenizer

        public StringTokenizer​(java.lang.String input,
                               char delim)
        Constructs a tokenizer splitting on the specified delimiter character.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter character
      • StringTokenizer

        public StringTokenizer​(java.lang.String input,
                               char delim,
                               char quote)
        Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter character
        quote - the field quoted string character
      • StringTokenizer

        public StringTokenizer​(java.lang.String input,
                               java.lang.String delim)
        Constructs a tokenizer splitting on the specified delimiter string.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter string
      • StringTokenizer

        public StringTokenizer​(java.lang.String input,
                               StringMatcher delim)
        Constructs a tokenizer splitting using the specified delimiter matcher.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter matcher
      • StringTokenizer

        public StringTokenizer​(java.lang.String input,
                               StringMatcher delim,
                               StringMatcher quote)
        Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter matcher
        quote - the field quoted string matcher
    • Method Detail

      • getCSVClone

        private static StringTokenizer getCSVClone()
        Returns a clone of CSV_TOKENIZER_PROTOTYPE.
        Returns:
        a clone of CSV_TOKENIZER_PROTOTYPE.
      • getCSVInstance

        public static StringTokenizer getCSVInstance()
        Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

        You must call a "reset" method to set the string which you want to parse.

        Returns:
        a new tokenizer instance which parses Comma Separated Value strings
      • getCSVInstance

        public static StringTokenizer getCSVInstance​(char[] input)
        Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).
        Parameters:
        input - the text to parse
        Returns:
        a new tokenizer instance which parses Comma Separated Value strings
      • getCSVInstance

        public static StringTokenizer getCSVInstance​(java.lang.String input)
        Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).
        Parameters:
        input - the text to parse
        Returns:
        a new tokenizer instance which parses Comma Separated Value strings
      • getTSVClone

        private static StringTokenizer getTSVClone()
        Returns a clone of TSV_TOKENIZER_PROTOTYPE.
        Returns:
        a clone of TSV_TOKENIZER_PROTOTYPE.
      • getTSVInstance

        public static StringTokenizer getTSVInstance()
        Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

        You must call a "reset" method to set the string which you want to parse.

        Returns:
        a new tokenizer instance which parses Tab Separated Value strings.
      • getTSVInstance

        public static StringTokenizer getTSVInstance​(char[] input)
        Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).
        Parameters:
        input - the string to parse
        Returns:
        a new tokenizer instance which parses Tab Separated Value strings.
      • getTSVInstance

        public static StringTokenizer getTSVInstance​(java.lang.String input)
        Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).
        Parameters:
        input - the string to parse
        Returns:
        a new tokenizer instance which parses Tab Separated Value strings.
      • add

        public void add​(java.lang.String obj)
        Unsupported ListIterator operation.
        Specified by:
        add in interface java.util.ListIterator<java.lang.String>
        Parameters:
        obj - this parameter ignored.
        Throws:
        java.lang.UnsupportedOperationException - always
      • addToken

        private void addToken​(java.util.List<java.lang.String> list,
                              java.lang.String tok)
        Adds a token to a list, paying attention to the parameters we've set.
        Parameters:
        list - the list to add to
        tok - the token to add
      • checkTokenized

        private void checkTokenized()
        Checks if tokenization has been done, and if not then do it.
      • clone

        public java.lang.Object clone()
        Creates a new instance of this Tokenizer. The new instance is reset so that it will be at the start of the token list. If a CloneNotSupportedException is caught, return null.
        Overrides:
        clone in class java.lang.Object
        Returns:
        a new instance of this Tokenizer which has been reset.
      • cloneReset

        java.lang.Object cloneReset()
                             throws java.lang.CloneNotSupportedException
        Creates a new instance of this Tokenizer. The new instance is reset so that it will be at the start of the token list.
        Returns:
        a new instance of this Tokenizer which has been reset.
        Throws:
        java.lang.CloneNotSupportedException - if there is a problem cloning
      • getContent

        public java.lang.String getContent()
        Gets the String content that the tokenizer is parsing.
        Returns:
        The string content being parsed
      • getDelimiterMatcher

        public StringMatcher getDelimiterMatcher()
        Gets the field delimiter matcher.
        Returns:
        The delimiter matcher in use
      • getIgnoredMatcher

        public StringMatcher getIgnoredMatcher()
        Gets the ignored character matcher.

        These characters are ignored when parsing the String, unless they are within a quoted region. The default value is not to ignore anything.

        Returns:
        The ignored matcher in use
      • getQuoteMatcher

        public StringMatcher getQuoteMatcher()
        Gets the quote matcher currently in use.

        The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data. The default value is '"' (double quote).

        Returns:
        The quote matcher in use
      • getTokenArray

        public java.lang.String[] getTokenArray()
        Gets a copy of the full token list as an independent modifiable array.
        Returns:
        The tokens as a String array
      • getTokenList

        public java.util.List<java.lang.String> getTokenList()
        Gets a copy of the full token list as an independent modifiable list.
        Returns:
        The tokens as a String array
      • getTrimmerMatcher

        public StringMatcher getTrimmerMatcher()
        Gets the trimmer character matcher.

        These characters are trimmed off on each side of the delimiter until the token or quote is found. The default value is not to trim anything.

        Returns:
        The trimmer matcher in use
      • hasNext

        public boolean hasNext()
        Tests whether there are any more tokens.
        Specified by:
        hasNext in interface java.util.Iterator<java.lang.String>
        Specified by:
        hasNext in interface java.util.ListIterator<java.lang.String>
        Returns:
        true if there are more tokens
      • hasPrevious

        public boolean hasPrevious()
        Tests whether there are any previous tokens that can be iterated to.
        Specified by:
        hasPrevious in interface java.util.ListIterator<java.lang.String>
        Returns:
        true if there are previous tokens
      • isEmptyTokenAsNull

        public boolean isEmptyTokenAsNull()
        Tests whether the tokenizer currently returns empty tokens as null. The default for this property is false.
        Returns:
        true if empty tokens are returned as null
      • isIgnoreEmptyTokens

        public boolean isIgnoreEmptyTokens()
        Tests whether the tokenizer currently ignores empty tokens. The default for this property is true.
        Returns:
        true if empty tokens are not returned
      • isQuote

        private boolean isQuote​(char[] srcChars,
                                int pos,
                                int len,
                                int quoteStart,
                                int quoteLen)
        Tests if the characters at the index specified match the quote already matched in readNextToken().
        Parameters:
        srcChars - the character array being tokenized
        pos - the position to check for a quote
        len - the length of the character array being tokenized
        quoteStart - the start position of the matched quote, 0 if no quoting
        quoteLen - the length of the matched quote, 0 if no quoting
        Returns:
        true if a quote is matched
      • next

        public java.lang.String next()
        Gets the next token.
        Specified by:
        next in interface java.util.Iterator<java.lang.String>
        Specified by:
        next in interface java.util.ListIterator<java.lang.String>
        Returns:
        The next String token
        Throws:
        java.util.NoSuchElementException - if there are no more elements
      • nextIndex

        public int nextIndex()
        Gets the index of the next token to return.
        Specified by:
        nextIndex in interface java.util.ListIterator<java.lang.String>
        Returns:
        The next token index
      • nextToken

        public java.lang.String nextToken()
        Gets the next token from the String. Equivalent to next() except it returns null rather than throwing NoSuchElementException when no tokens remain.
        Returns:
        The next sequential token, or null when no more tokens are found
      • previous

        public java.lang.String previous()
        Gets the token previous to the last returned token.
        Specified by:
        previous in interface java.util.ListIterator<java.lang.String>
        Returns:
        The previous token
      • previousIndex

        public int previousIndex()
        Gets the index of the previous token.
        Specified by:
        previousIndex in interface java.util.ListIterator<java.lang.String>
        Returns:
        The previous token index
      • previousToken

        public java.lang.String previousToken()
        Gets the previous token from the String.
        Returns:
        The previous sequential token, or null when no more tokens are found
      • readNextToken

        private int readNextToken​(char[] srcChars,
                                  int start,
                                  int len,
                                  TextStringBuilder workArea,
                                  java.util.List<java.lang.String> tokenList)
        Reads character by character through the String to get the next token.
        Parameters:
        srcChars - the character array being tokenized
        start - the first character of field
        len - the length of the character array being tokenized
        workArea - a temporary work area
        tokenList - the list of parsed tokens
        Returns:
        The starting position of the next field (the character immediately after the delimiter), or -1 if end of string found
      • readWithQuotes

        private int readWithQuotes​(char[] srcChars,
                                   int start,
                                   int len,
                                   TextStringBuilder workArea,
                                   java.util.List<java.lang.String> tokenList,
                                   int quoteStart,
                                   int quoteLen)
        Reads a possibly quoted string token.
        Parameters:
        srcChars - the character array being tokenized
        start - the first character of field
        len - the length of the character array being tokenized
        workArea - a temporary work area
        tokenList - the list of parsed tokens
        quoteStart - the start position of the matched quote, 0 if no quoting
        quoteLen - the length of the matched quote, 0 if no quoting
        Returns:
        The starting position of the next field (the character immediately after the delimiter, or if end of string found, then the length of string
      • remove

        public void remove()
        Throws UnsupportedOperationException for this unsupported ListIterator operation.
        Specified by:
        remove in interface java.util.Iterator<java.lang.String>
        Specified by:
        remove in interface java.util.ListIterator<java.lang.String>
        Throws:
        java.lang.UnsupportedOperationException - always
      • reset

        public StringTokenizer reset()
        Resets this tokenizer, forgetting all parsing and iteration already completed.

        This method allows the same tokenizer to be reused for the same String.

        Returns:
        this, to enable chaining
      • reset

        public StringTokenizer reset​(char[] input)
        Resets this tokenizer, giving it a new input string to parse. In this manner you can re-use a tokenizer with the same settings on multiple input lines.
        Parameters:
        input - the new character array to tokenize, not cloned, null sets no text to parse
        Returns:
        this, to enable chaining
      • reset

        public StringTokenizer reset​(java.lang.String input)
        Resets this tokenizer, giving it a new input string to parse. In this manner you can re-use a tokenizer with the same settings on multiple input lines.
        Parameters:
        input - the new string to tokenize, null sets no text to parse
        Returns:
        this, to enable chaining
      • set

        public void set​(java.lang.String obj)
        Throws UnsupportedOperationException for this unsupported ListIterator operation.
        Specified by:
        set in interface java.util.ListIterator<java.lang.String>
        Parameters:
        obj - this parameter ignored.
        Throws:
        java.lang.UnsupportedOperationException - always
      • setDelimiterChar

        public StringTokenizer setDelimiterChar​(char delim)
        Sets the field delimiter character.
        Parameters:
        delim - the delimiter character to use
        Returns:
        this, to enable chaining
      • setDelimiterMatcher

        public StringTokenizer setDelimiterMatcher​(StringMatcher delim)
        Sets the field delimiter matcher.

        The delimiter is used to separate one token from another.

        Parameters:
        delim - the delimiter matcher to use
        Returns:
        this, to enable chaining
      • setDelimiterString

        public StringTokenizer setDelimiterString​(java.lang.String delim)
        Sets the field delimiter string.
        Parameters:
        delim - the delimiter string to use
        Returns:
        this, to enable chaining
      • setEmptyTokenAsNull

        public StringTokenizer setEmptyTokenAsNull​(boolean emptyAsNull)
        Sets whether the tokenizer should return empty tokens as null. The default for this property is false.
        Parameters:
        emptyAsNull - whether empty tokens are returned as null
        Returns:
        this, to enable chaining
      • setIgnoredChar

        public StringTokenizer setIgnoredChar​(char ignored)
        Sets the character to ignore.

        This character is ignored when parsing the String, unless it is within a quoted region.

        Parameters:
        ignored - the ignored character to use
        Returns:
        this, to enable chaining
      • setIgnoredMatcher

        public StringTokenizer setIgnoredMatcher​(StringMatcher ignored)
        Sets the matcher for characters to ignore.

        These characters are ignored when parsing the String, unless they are within a quoted region.

        Parameters:
        ignored - the ignored matcher to use, null ignored
        Returns:
        this, to enable chaining
      • setIgnoreEmptyTokens

        public StringTokenizer setIgnoreEmptyTokens​(boolean ignoreEmptyTokens)
        Sets whether the tokenizer should ignore and not return empty tokens. The default for this property is true.
        Parameters:
        ignoreEmptyTokens - whether empty tokens are not returned
        Returns:
        this, to enable chaining
      • setQuoteChar

        public StringTokenizer setQuoteChar​(char quote)
        Sets the quote character to use.

        The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data.

        Parameters:
        quote - the quote character to use
        Returns:
        this, to enable chaining
      • setQuoteMatcher

        public StringTokenizer setQuoteMatcher​(StringMatcher quote)
        Set the quote matcher to use.

        The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data.

        Parameters:
        quote - the quote matcher to use, null ignored
        Returns:
        this, to enable chaining
      • setTrimmerMatcher

        public StringTokenizer setTrimmerMatcher​(StringMatcher trimmer)
        Sets the matcher for characters to trim.

        These characters are trimmed off on each side of the delimiter until the token or quote is found.

        Parameters:
        trimmer - the trimmer matcher to use, null ignored
        Returns:
        this, to enable chaining
      • size

        public int size()
        Gets the number of tokens found in the String.
        Returns:
        The number of matched tokens
      • tokenize

        protected java.util.List<java.lang.String> tokenize​(char[] srcChars,
                                                            int offset,
                                                            int count)
        Internal method to performs the tokenization.

        Most users of this class do not need to call this method. This method will be called automatically by other (public) methods when required.

        This method exists to allow subclasses to add code before or after the tokenization. For example, a subclass could alter the character array, offset or count to be parsed, or call the tokenizer multiple times on multiple strings. It is also be possible to filter the results.

        StrTokenizer will always pass a zero offset and a count equal to the length of the array to this method, however a subclass may pass other values, or even an entirely different array.

        Parameters:
        srcChars - the character array being tokenized, may be null
        offset - the start position within the character array, must be valid
        count - the number of characters to tokenize, must be valid
        Returns:
        The modifiable list of String tokens, unmodifiable if null array or zero count
      • toString

        public java.lang.String toString()
        Gets the String content that the tokenizer is parsing.
        Overrides:
        toString in class java.lang.Object
        Returns:
        The string content being parsed