Java StringTokenizer - Keep Delimiters

Java StringTokenizer – Keep Delimiters example shows how to keep delimiters while tokenizing the string content using the StringTokenizer object.

How to keep delimiters while using the StringTokenizer object?

In some cases, we want to get the delimiters as tokens as well while using the StringTokenizer object to tokenizing the string content. When we create the StringTokenizer object using the one or two-argument constructors, it does not return the delimiters as tokens. Please see the below-given code example to understand that.

package com.javacodeexamples.stringtokenizerexamples;

import java.util.StringTokenizer;

public class StringTokenizerKeepDelimiters {

public static void main(String[] args) {

String strContent = "one,two,three,four";

StringTokenizer st = new StringTokenizer(strContent, ",");

while(st.hasMoreTokens()) {

System.out.println(st.nextToken());

}

Output

one

two

three

four

As you can see from the output, the delimiter “,” is not returned back as tokens. In order to keep the delimiters as well, we need to use the below-given constructor while creating the StringTokenizer object.

1	public StringTokenizer(String string, String delimiter, boolean returnDelimiters)

If we pass true as the third argument, i.e. returnDelimiters, the StringTokenizer object also returns the delimiters as tokens when we iterate through the tokens.

See below code example showing that behavior.

package com.javacodeexamples.stringtokenizerexamples;

import java.util.StringTokenizer;

public class StringTokenizerKeepDelimiters {

public static void main(String[] args) {

String strContent = "one,two,three,four";

StringTokenizer st = new StringTokenizer(strContent, ",", true);

while(st.hasMoreTokens()) {

System.out.println(st.nextToken());

}

Output

one

two

three

four

By default, the StringTokenizer object does not return the empty or blank tokens. In that particular case, getting delimiters as tokens allows us to determine if the token is empty or blank. (If we get two consecutive delimiter tokens, it means that the token is empty or blank).

If you want to learn more about the tokenizer, please visit the Java StringTokenizer tutorial.

Please let me know your views in the comments section below.