Text, Unicode and regular expressions

Text is the part of a program where a test most often passes for the wrong reason: a trailing space, a carriage return or a character that looks like a letter but is not. This lesson tests a small text helper with four cases, one of them a Unicode word, and shows where the invisible characters live.

A text helper

Text.java has two methods. normalize turns Windows line endings into Unix ones and strips the edges; isWord answers whether the whole string is letters, using the Unicode letter class rather than [A-Za-z].

public class Text { static String normalize(String text) { return text.replace("\r\n", "\n").strip(); } static boolean isWord(String text) { return text.matches("\\p{L}+"); } }

The Java source writes \p{L}+: one backslash starts the escape and the second makes it a literal backslash for the regular expression engine. A page written in a PHP single-quoted string needs four backslashes to produce those two, which is why the block above looks longer than the Java file.

Strip and line endings

strip removes white space from both ends, including the space, the tab and the line terminators. It does not touch the middle. The replace runs first, so a Windows file becomes the Unix form before the edges go. The order matters: stripping first would leave the carriage return of the last line in the middle of the text.

Unicode letters

[A-Za-z] is the Latin alphabet only; \p{L} is every Unicode letter, which includes ï and the Cyrillic letters of привет. A test that says a word is a word should pass for both, and the two assertions in one test say so. A word with a digit is not a word under this rule, which is the fourth test.

The regular expression

String.matches anchors the whole string: it is true only when the entire input matches the pattern, which is why the method needs no ^ or $. The + requires at least one letter, so the empty string is not a word.

import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; import org.junit.jupiter.api.Test; public class TextTest { @Test void stripsTheEdges() { assertEquals("Ada", Text.normalize(" Ada\n")); } @Test void turnsWindowsEndingsIntoUnixOnes() { assertEquals("a\nb", Text.normalize("a\r\nb")); } @Test void acceptsUnicodeLetters() { assertTrue(Text.isWord("naïve")); assertTrue(Text.isWord("привет")); } @Test void rejectsDigits() { assertFalse(Text.isWord("abc123")); } }

The report

javac -Xlint:all -cp /opt/java-tools/junit-platform-console-standalone.jar -d out Text.java TextTest.java
junit execute --class-path out --scan-classpath --disable-ansi-colors

Thanks for using JUnit! Support its development at https://junit.org/sponsoring

╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
│  └─ TextTest ✔
│     ├─ acceptsUnicodeLetters() ✔
│     ├─ stripsTheEdges() ✔
│     ├─ rejectsDigits() ✔
│     └─ turnsWindowsEndingsIntoUnixOnes() ✔
└─ JUnit Vintage ✔

Test run finished after 881 ms
[         4 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         4 containers successful ]
[         0 containers failed     ]
[         4 tests found           ]
[         0 tests skipped         ]
[         4 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         0 tests failed          ]

The invisible character

When a text assertion fails, the message prints the two strings between angle brackets, and a space or a carriage return is invisible. The failure of assertEquals("Ada", "Ada ") looks like two equal words. Read the lengths before the values when a text failure makes no sense, and print the code points of the characters around the difference. The report below the tree is the place to look; the tree line is only the first sentence.

A page has the same problem. A trailing space in a code block is easy to lose and impossible to see, which is why the course writes the output blocks with   where the column must stay.

Exercises

  1. Add a test for a string that is only spaces and decide what normalize should return.
  2. Change isWord to use [A-Za-z]+ and run. Which assertion fails?
  3. Add a test for an empty string and read the result. Is it a word under the current rule?
  4. Print the length of the two strings in a failing assertion and compare it with what the report shows.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does String.strip remove?

    Answer

    White space at both ends, including line terminators

    strip trims both ends and understands the Unicode white-space set.

  2. What does \\p{L} match in a Java regular expression?

    Answer

    Any Unicode letter

    The property L is the Unicode letter category, so accented and Cyrillic letters match.

  3. Why does String.matches not need ^ and $?

    Answer

    It matches the whole string by contract

    matches requires the entire input to match the pattern.

  4. What does this program print? public class Q1 { public static void main(String[] args) { System.out.println(" Ada ".strip()); } }

    Answer

    Ada

    strip removes the spaces at both ends.

  5. What does this program print? public class Q2 { public static void main(String[] args) { System.out.println("naïve".length()); } }

    Answer

    5

    The string has five characters, including the accented i.

  6. What does this program print? public class Q3 { public static void main(String[] args) { System.out.println("abc123".matches("[a-z]+")); } }

    Answer

    false

    The digits break the match, so the whole string does not match.

  7. What does this program print? public class Q4 { public static void main(String[] args) { System.out.println("a\r\nb".lines().count()); } }

    Answer

    2

    The Windows line ending is one terminator, so there are two lines.

  8. Why is a text failure with an invisible character hard to read?

    Answer

    The report prints the strings, and the difference is a character that does not show

    A trailing space or a carriage return looks like nothing between the angle brackets.

  9. What does this program print? public class Q5 { public static void main(String[] args) { System.out.println("abc".matches("\\p{L}+")); } }

    Answer

    true

    All three characters are letters, so the whole string matches the letter class.

  10. Why does the page write four backslashes for \\p{L}+?

    Answer

    The PHP single-quoted string halves them once more than the Java source needs

    PHP reduces \\ to \, so four become the two the Java source needs.

The code on this page was built with OpenJDK 21.0.13 (javac) and run with JUnit Platform Console Launcher 1.14.3 on Alpine Linux in a Docker container.

Article author: Arthur Isaev