The StringTokenizer Class

The java.util.StringTokenizer class allows an application to break a string into individual pieces, called tokens.

This class is considered a legacy class, and while it is still supported for compatibility reasons, modern Java development generally prefers using the String.split() method (from the java.lang package) or the java.util.regex package. However, you will often encounter StringTokenizer in older codebases.

---

How does it work?

A StringTokenizer object maintains a current position within the string to be tokenized. It advances through the string, returning the characters between delimiters (the separator characters).

The Constructors

1. StringTokenizer(String str): Creates a StringTokenizer with default delimiters (space, tab, newline, carriage return, and form feed).
2. StringTokenizer(String str, String delim): Creates a StringTokenizer with a specified set of delimiters.

Key Methods

• hasMoreTokens(): Checks if there are more tokens available in the string. Returns true or false.
• nextToken(): Returns the next token from the string.

---

Example: Default Delimiters

By default, the StringTokenizer splits strings using spaces and tabs.

import java.util.StringTokenizer;

public class LabTokenizer1 {

  public static void main(String args[]) {
    String sentence = "Java is an amazing programming language";

    StringTokenizer st = new StringTokenizer(sentence);

    // Loop as long as there are more tokens
    while (st.hasMoreTokens()) {
      System.out.println(st.nextToken());
    }
  }
}

Output:

Java
is
an
amazing
programming
language

---

Example: Custom Delimiters

You can specify a custom string of delimiters. The tokenizer will split the string wherever it finds any single character from the delimiter string.

import java.util.StringTokenizer;

public class LabTokenizer2 {

  public static void main(String args[]) {
    // A comma-separated string
    String data = "apple,banana,orange,grape";

    // Specifying the comma as the delimiter
    StringTokenizer st = new StringTokenizer(data, ",");

    while (st.hasMoreTokens()) {
      System.out.println("Fruit: " + st.nextToken());
    }
  }
}

Output:

Fruit: apple
Fruit: banana
Fruit: orange
Fruit: grape

[!WARNING] If you pass ",;" as the delimiter, the tokenizer will split the string at every comma and at every semicolon. It does not look for the literal string ",;". If you need to split by complex literal strings or regular expressions, use String.split() instead!