Java StringTokenizer Class

Last Updated : 10 Jan, 2025

StringTokenizer class in Java is used to break a string into tokens based on delimiters. A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the characters processed.

A token is returned by taking a substring of the string that was used to create the StringTokenizer object.
It provides the first step in the parsing process often called lexer or scanner.
It implements the Enumeration interface.
To perform Java String Tokenization, we need to specify an input string and a set of delimiters.
A delimiter is a character or set of characters that separate tokens in the string.

Note: StringTokenizer is a legacy class, and the split() method is preferred for modern applications.

Example: Below is a simple example that explains the use of Java StringTokenizer to split a space-separated string into tokens:

Java

// Demonstration of Java StringTokenizer
import java.util.StringTokenizer;

public class Geeks {
    public static void main(String[] args) {
      
        // Input string
        String s = "Hello Geeks how are you";

        // Create a StringTokenizer object 
        // with space as the delimiter
        StringTokenizer st = new StringTokenizer(s, " ");

        // Tokenize the string and print each token
        while (st.hasMoreTokens()) {
            System.out.println(st.nextToken());
        }
    }
}

Output

Hello
Geeks
how
are
you

Explanation: In the above example, we have created a StringTokenizer object by passing the string and a space " " as the delimiter. The hasMoreTokens() method checks there are more tokens available to process or not. The nextToken() method get the next token (substring).

Below is the representation of the process, which we defined in the above example:

StringTokenizer Class Representation in Java

Constructors of StringTokenizer Class

The StringTokenizer class provides three constructors to tokenize strings in different ways.

Constructors	Description
StringTokenizer(String str)	Creates a tokenizer for the specified string. Uses default delimiters (whitespace, tabs, etc.).
StringTokenizer(String str, String delim)	Creates a tokenizer for the specified string using the given delimiters.
StringTokenizer(String str, String delim, boolean returnDelims)	Creates a tokenizer for the specified string using the given delimiters and specifies whether the delimiters should be returned as tokens.

Note:

Default Delimiters: When no delimiter is specified, whitespace is used.
returnDelims: If true, the delimiters are treated as tokens themselves.

Below is a concise explanation of how each constructor works, along with a code example in the combined way.

Cases of StringTokenizer Constructors

1. If the returnDelims is false, delimiter characters serve to separate tokens.

Example:

Input: if string --> "hello geeks" and Delimiter is " ", then
Output: tokens are "hello" and "geeks".

2. If the returnDelims is true, delimiter characters are considered to be tokens.

Example:

Input: String --> is "hello geeks" and Delimiter is " ", then
Output: Tokens --> "hello", " " and "geeks".

3. Multiple delimiters can be chosen for a single string.

Example:

Syntax: StringTokenizer st1 = new StringTokenizer( "2+3-1*8/4", "+*-/");

Input: String --> is "2+3-1*8/4" and Delimiters are +,*,-,/
Output: Tokens --> "2","3","1","8","4".

Example:

Java

// Demonstration of String Tokenizer Constructors
import java.util.*;

class Geeks {

    public static void main(String[] args) {
      
        // Example with Constructor 1
        System.out.println("Using StringTokenizer Constructor 1: ");

        // Using StringTokenizer to split the string into 
        // tokens using space (" ") as the delimiter
        StringTokenizer st1 = new StringTokenizer(
            "Geeks fo Geeks", " ");

        // Iterate through tokens while 
        // there are more tokens available
        while (st1.hasMoreTokens())
          
            // Getting and printing the next token
            System.out.println(st1.nextToken());

        // Example with Constructor 2
        System.out.println("Using StringTokenizer Constructor 2: ");

        // Using StringTokenizer to split the string 
        // using ":" as the delimiter
        StringTokenizer st2 = new StringTokenizer(
            "java : Code : String : Tokenizer", " :");

        // Iterate through tokens and print them
        while (st2.hasMoreTokens())
            System.out.println(st2.nextToken());

        // Example with Constructor 3
        System.out.println("Using StringTokenizer Constructor 3: ");

        // Using StringTokenizer with returnDelims = true 
        // to include delimiters as tokens
        StringTokenizer st3 = new StringTokenizer(
            "java : Code", " :", true);

        // Iterate through tokens (including delimiters) 
        // and print them
        while (st3.hasMoreTokens())
            System.out.println(st3.nextToken());
    }
}

Output

Using StringTokenizer Constructor 1: 
Geeks
fo
Geeks
Using StringTokenizer Constructor 2: 
java
Code
String
Tokenizer
Using StringTokenizer Constructor 3: 
java
 
:
 
Code

Methods Of StringTokenizer Class

Below are some commonly used methods of StringTokenizer class along with a combined code example demonstrating some of these methods.

Method	Action Performed
countTokens()	Returns the total number of tokens present.
hasMoreTokens()	Tests if tokens are present for the StringTokenizer's string.
nextElement()	Returns an Object rather than String.
hasMoreElements()	Returns the same value as hasMoreToken.
nextToken()	Returns the next token from the given StringTokenizer.

Example:

Java

// Demonstration of StringTokenizer Methods
import java.util.*;

class Geeks {
    public static void main(String[] args) {
      
        // Creating a StringTokenizer
        StringTokenizer st = new StringTokenizer(
            "Welcome to GeeksforGeeks");

        StringTokenizer st1 = new StringTokenizer("");

          // countTokens Method
        int c = st.countTokens();
        System.out.println(c);
      
          // hasMoreTokens Methods
          System.out.println("Welcome to GeeksforGeeks: "+ st.hasMoreTokens());
          System.out.println("(Empty String) : "+ st1.hasMoreTokens());
      
          // nextElement() Method
          System.out.println("\nTraversing the String:");
      
          while(st.hasMoreTokens()){
              System.out.println(st.nextElement());
        }
          
    }
}

Output

3
Welcome to GeeksforGeeks: true
(Empty String) : false

Traversing the String:
Welcome
to
GeeksforGeeks

Java StringTokenizer Class

Mohit Gupta

Improve

Article Tags :

Practice Tags :

Java StringTokenizer Class

Constructors of StringTokenizer Class

Cases of StringTokenizer Constructors

Methods Of StringTokenizer Class

Similar Reads

Thank You!

What kind of Experience do you want to share?