Skip to content

Java StringTokenizer – Using RegEx Pattern

Java StringTokenizer – Using RegEx Pattern example shows what happens when we use regex pattern as delimiter to tokenize the string.

Can we use a regex pattern as delimiter to tokenize the string?

The StringTokenizer class in Java is used to tokenize the string content. We can provide the set of delimiters using which we want to generate the tokens from the given string content.

One way to do this is to create a StringTokenizer object using the below given constructor.

This constructor creates a new StringTokenizer object that will tokenize the string content based on the set of delimiters we passed. Once the object is created, we can get the tokens from the string using hasMoreTokens and nextToken methods as given in below example.


In the above example, I used a simple space character to create tokens from the string. However, many a time we need more complex approach to generate tokens, for example, using a regex pattern.

However, the StringTokenizer does not support regex pattern. Providing a regex pattern as delimiters does not work as given below.


As you can see from the output, the word boundary “\\b” regex pattern did not work and tokens were not generated.

How to generate tokens based on regex pattern?

The StringTokenizer does not support the regex pattern, but you can use the String split method to do the same. The only difference is, the split method returns an array containing all tokens.

Here is the same example using the split method.


If you want to tokenize the string using a regex pattern, just use the split method instead of the StringTokenizer class.

If you want to learn more about the string tokenizer, please visit the Java StringTokenizer Tutorial.

Please let me know your views in the comments section below.

About the author

Leave a Reply

Your email address will not be published.