Java StringTokenizer Class
Last Updated :
10 Jan, 2025
StringTokenizer class in Java is used to break a string into tokens based on delimiters. A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the characters processed.
- A token is returned by taking a substring of the string that was used to create the StringTokenizer object.
- It provides the first step in the parsing process often called lexer or scanner.
- It implements the
Enumeration interface. - To perform Java String Tokenization, we need to specify an input string and a set of delimiters.
- A delimiter is a character or set of characters that separate tokens in the string.
Note: StringTokenizer is a legacy class, and the split() method is preferred for modern applications.
Example: Below is a simple example that explains the use of Java StringTokenizer to split a space-separated string into tokens:
Java
// Demonstration of Java StringTokenizer
import java.util.StringTokenizer;
public class Geeks {
public static void main(String[] args) {
// Input string
String s = "Hello Geeks how are you";
// Create a StringTokenizer object
// with space as the delimiter
StringTokenizer st = new StringTokenizer(s, " ");
// Tokenize the string and print each token
while (st.hasMoreTokens()) {
System.out.println(st.nextToken());
}
}
}
OutputHello
Geeks
how
are
you
Explanation: In the above example, we have created a StringTokenizer object by passing the string and a space " " as the delimiter. The hasMoreTokens() method checks there are more tokens available to process or not. The nextToken() method get the next token (substring).
Below is the representation of the process, which we defined in the above example:
Constructors of StringTokenizer Class
The StringTokenizer class provides three constructors to tokenize strings in different ways.
Constructors | Description |
|---|
StringTokenizer(String str) | Creates a tokenizer for the specified string. Uses default delimiters (whitespace, tabs, etc.). |
|---|
StringTokenizer(String str, String delim) | Creates a tokenizer for the specified string using the given delimiters. |
|---|
StringTokenizer(String str, String delim, boolean returnDelims) | Creates a tokenizer for the specified string using the given delimiters and specifies whether the delimiters should be returned as tokens. |
|---|
Note:
- Default Delimiters: When no delimiter is specified, whitespace is used.
- returnDelims: If true, the delimiters are treated as tokens themselves.
Below is a concise explanation of how each constructor works, along with a code example in the combined way.
Cases of StringTokenizer Constructors
1. If the returnDelims is false, delimiter characters serve to separate tokens.
Example:
Input: if string --> "hello geeks" and Delimiter is " ", then
Output: tokens are "hello" and "geeks".
2. If the returnDelims is true, delimiter characters are considered to be tokens.
Example:
Input: String --> is "hello geeks" and Delimiter is " ", then
Output: Tokens --> "hello", " " and "geeks".
3. Multiple delimiters can be chosen for a single string.
Example:
Syntax: StringTokenizer st1 = new StringTokenizer( "2+3-1*8/4", "+*-/");
Input: String --> is "2+3-1*8/4" and Delimiters are +,*,-,/
Output: Tokens --> "2","3","1","8","4".
Example:
Java
// Demonstration of String Tokenizer Constructors
import java.util.*;
class Geeks {
public static void main(String[] args) {
// Example with Constructor 1
System.out.println("Using StringTokenizer Constructor 1: ");
// Using StringTokenizer to split the string into
// tokens using space (" ") as the delimiter
StringTokenizer st1 = new StringTokenizer(
"Geeks fo Geeks", " ");
// Iterate through tokens while
// there are more tokens available
while (st1.hasMoreTokens())
// Getting and printing the next token
System.out.println(st1.nextToken());
// Example with Constructor 2
System.out.println("Using StringTokenizer Constructor 2: ");
// Using StringTokenizer to split the string
// using ":" as the delimiter
StringTokenizer st2 = new StringTokenizer(
"java : Code : String : Tokenizer", " :");
// Iterate through tokens and print them
while (st2.hasMoreTokens())
System.out.println(st2.nextToken());
// Example with Constructor 3
System.out.println("Using StringTokenizer Constructor 3: ");
// Using StringTokenizer with returnDelims = true
// to include delimiters as tokens
StringTokenizer st3 = new StringTokenizer(
"java : Code", " :", true);
// Iterate through tokens (including delimiters)
// and print them
while (st3.hasMoreTokens())
System.out.println(st3.nextToken());
}
}
OutputUsing StringTokenizer Constructor 1:
Geeks
fo
Geeks
Using StringTokenizer Constructor 2:
java
Code
String
Tokenizer
Using StringTokenizer Constructor 3:
java
:
Code
Methods Of StringTokenizer Class
Below are some commonly used methods of StringTokenizer class along with a combined code example demonstrating some of these methods.
Example:
Java
// Demonstration of StringTokenizer Methods
import java.util.*;
class Geeks {
public static void main(String[] args) {
// Creating a StringTokenizer
StringTokenizer st = new StringTokenizer(
"Welcome to GeeksforGeeks");
StringTokenizer st1 = new StringTokenizer("");
// countTokens Method
int c = st.countTokens();
System.out.println(c);
// hasMoreTokens Methods
System.out.println("Welcome to GeeksforGeeks: "+ st.hasMoreTokens());
System.out.println("(Empty String) : "+ st1.hasMoreTokens());
// nextElement() Method
System.out.println("\nTraversing the String:");
while(st.hasMoreTokens()){
System.out.println(st.nextElement());
}
}
}
Output3
Welcome to GeeksforGeeks: true
(Empty String) : false
Traversing the String:
Welcome
to
GeeksforGeeks
Similar Reads
Java Reader Class Reader class in Java is an abstract class used for reading character streams. It serves as the base class for various subclasses like FileReader, BufferedReader, CharArrayReader, and others, which provide more efficient implementations of the read() method. To work with the Reader class, we must ext
6 min read
Java Pattern Class The Pattern class in Java is used for defining regular expressions (regex) to perform pattern matching on strings. It is part of the java.util.regex package and it plays a key role in searching, replacing, and manipulating strings based on patterns. The Matcher class works together with Pattern to p
3 min read
Java.io.StreamTokenizer Class in Java | Set 2 StringTokenizer Class in Java | Set 1 Methods: parseNumbers() : java.io.StreamTokenizer.parseNumbers() specifies that the number in StreamTokenizer is parsed, so that each character - " 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 " has a numeric attribute. When the parser encounters a word token that has the forma
8 min read
Java.io.StreamTokenizer Class in Java | Set 1 In Java, the StreamTokenizer class is present in the java.io package. It is used to parse an input stream by breaking it into small chunks known as tokens, these tokens make the processing easier. A token can be a word, a number, or any specific symbol. Stream Tokenizer can recognize numbers, quoted
12 min read
StringTokenizer hasMoreTokens() Method in Java with Examples The hasMoreTokens() method of StringTokenizer class checks whether there are any more tokens available with this StringTokenizer. Syntax: public boolean hasMoreTokens() Parameters: The method does not take any parameters. Return Value: The method returns boolean True if the availability of at least
2 min read
Matcher Class in Java In Java, Matcher is a class that is implemented by the MatchResult interface, that performs match operations on a character sequence by interpreting a Pattern. Below, we can see the declaration of java.util.regex.Matcher in java.lang.Object Class: public final class Matcher extends Object implements
4 min read
StringTokenizer countTokens() Method in Java with Examples The countTokens() method of StringTokenizer class calculate the number of times that this tokenizer's nextToken method can be called before the method generates any further exception. Note: The current position is not advanced during the process. Syntax: public int countTokens() Parameters: The meth
2 min read
StringTokenizer hasMoreElements() Method in Java with Examples The hasMoreElements() method of StringTokenizer class also checks whether there are any more tokens available with this StringTokenizer. It is similar to the hasMoreTokens(). The method exists exclusively so that the Enumeration interface of this class can be implemented. Syntax: public boolean hasM
2 min read
StringTokenizer nextToken() Method in Java with Examples The nextToken() method of StringTokenizer class is used to return the next token one after another from this StringTokenizer. Syntax: public String nextToken() Parameters: The method does not take any parameters. Return Value: The method returns the next token present in the line of the string token
1 min read
StringTokenizer nextElement() Method in Java with Examples The nextElement() method of StringTokenizer class is also used to return the next token one after another from this StringTokenizer. It is similar to the nextToken() method, except that the return type is Object rather than the String. Syntax: public Object nextElement() Parameters: The method does
2 min read