Assertions and a test driver in Haskell

A test is a small program that compares what a function returns with what it should return, and the whole value of a suite is in one bit: did the run pass. Haskell has good test libraries, and the ones people reach for first, HUnit and hspec, are not in this image: there is no network here to fetch them. That is not a reason to ship untested code. The three parts of a suite, a name for each check, a message for the failure and an exit status for the run, fit in thirty lines, and those thirty lines are what the libraries wrap.

This page builds that driver, uses it on a small module with two functions, and then breaks two of the expectations on purpose to show the two things a suite has to get right: every failure is reported, and the run tells its caller that it failed.

Why a driver of your own

The contract of a test run is the exit status. A program that prints all tests passed and exits 0 is a suite that passed; a program that prints nothing, or prints what it likes and exits 0 anyway, is a suite that cannot fail, and nobody will notice when it stops checking anything. cabal test reads the status of the program it runs, and every continuous integration system reads it from there.

The second thing a suite needs is a line per check. A test that hides its name behind a stack trace is a test whose failure takes a minute to locate, and a suite that stops at the first failure hides the other three failures behind the same minute. The driver below prints ok or not ok with the name of the check, keeps going, and counts.

The assertion helper

A check is a name, a boolean and a message that is only filled in when the check fails. The message is built where the values are still in scope, because that is the only place that knows what was compared:

data Result = Result { resultName :: String , resultPassed :: Bool , resultDetail :: String } check :: String -> Bool -> Result check name passed = Result name passed "" checkEqual :: (Eq a, Show a) => String -> a -> a -> Result checkEqual name expected actual | expected == actual = Result name True "" | otherwise = Result name False (printf "expected %s, got %s" (show expected) (show actual))

checkEqual takes the expected value and the actual value, both of any type that can be compared and shown, which is what the Eq a, Show a constraint says. The message is built in the otherwise branch only, so a suite of a thousand passing checks builds no messages at all.

The three field names of the record are also its accessor functions, which is why the driver reads resultPassed result without a match. The record has no deriving clause: it is never printed as a whole, and an instance nobody needs is code to maintain.

A program under test

The code under test is two functions with the kind of edge that a test should catch. clamp keeps a value inside a range with max and min, and parseCount reads a count from a line of input and rejects everything that is not digits.

The empty string is the case worth writing down: all isDigit "" is True, so without the null guard the function would call read "" and die with Prelude.read: no parse. A guard that is not tested is a guard that somebody deletes in a refactor.

The passing run

Six checks, the driver, and one program. Everything is in one file because the subject of the page is the driver; the next pages split the same code into a module and a suite.

module Main (main) where import Control.Monad (forM_) import Data.Char (isDigit) import Data.Maybe (isNothing) import System.Exit (exitFailure, exitSuccess) import Text.Printf (printf) data Result = Result { resultName :: String , resultPassed :: Bool , resultDetail :: String } check :: String -> Bool -> Result check name passed = Result name passed "" checkEqual :: (Eq a, Show a) => String -> a -> a -> Result checkEqual name expected actual | expected == actual = Result name True "" | otherwise = Result name False (printf "expected %s, got %s" (show expected) (show actual)) clamp :: Int -> Int -> Int -> Int clamp low high = max low . min high parseCount :: String -> Maybe Int parseCount text | not (null text) && all isDigit text = Just (read text) | otherwise = Nothing checks :: [Result] checks = [ checkEqual "clamp below the range" 1 (clamp 1 10 0) , checkEqual "clamp inside the range" 5 (clamp 1 10 5) , checkEqual "clamp above the range" 10 (clamp 1 10 99) , checkEqual "parse a number" (Just 42) (parseCount "42") , checkEqual "reject a word" Nothing (parseCount "four") , check "reject an empty string" (isNothing (parseCount "")) ] main :: IO () main = do forM_ (zip [1 :: Int ..] checks) $ \(number, result) -> if resultPassed result then printf "ok %d - %s\n" number (resultName result) else printf "not ok %d - %s: %s\n" number (resultName result) (resultDetail result) let failed = length (filter (not . resultPassed) checks) printf "%d checks, %d failed\n" (length checks) failed if failed == 0 then exitSuccess else exitFailure

The build is clean under -Wall and hlint reports No hints.

ghc -Wall -O1 -o checks Checks.hs
./checks

[1 of 2] Compiling Main ( Checks.hs, Checks.o )
[2 of 2] Linking checks
ok 1 - clamp below the range
ok 2 - clamp inside the range
ok 3 - clamp above the range
ok 4 - parse a number
ok 5 - reject a word
ok 6 - reject an empty string
6 checks, 0 failed

The status is the interesting part, and a shell shows it with $? right after the program:

./checks; echo "status $?"

ok 1 - clamp below the range
ok 2 - clamp inside the range
ok 3 - clamp above the range
ok 4 - parse a number
ok 5 - reject a word
ok 6 - reject an empty string
6 checks, 0 failed
status 0

exitSuccess and exitFailure from System.Exit are the two calls behind that number. They end the program immediately, which is why the summary line is printed before the call and not after it.

A failing check

Two expectations of the same program are wrong: the range starts at 1, not at 0, and parseCount "four" is Nothing, not Just 0. Both mistakes are the kind a copy-paste produces, and the driver reports both of them in one run.

module Main (main) where import Control.Monad (forM_) import Data.Char (isDigit) import Data.Maybe (isNothing) import System.Exit (exitFailure, exitSuccess) import Text.Printf (printf) data Result = Result { resultName :: String , resultPassed :: Bool , resultDetail :: String } check :: String -> Bool -> Result check name passed = Result name passed "" checkEqual :: (Eq a, Show a) => String -> a -> a -> Result checkEqual name expected actual | expected == actual = Result name True "" | otherwise = Result name False (printf "expected %s, got %s" (show expected) (show actual)) clamp :: Int -> Int -> Int -> Int clamp low high = max low . min high parseCount :: String -> Maybe Int parseCount text | not (null text) && all isDigit text = Just (read text) | otherwise = Nothing checks :: [Result] checks = [ checkEqual "clamp below the range" 0 (clamp 1 10 0) , checkEqual "clamp inside the range" 5 (clamp 1 10 5) , check "reject a word" (isNothing (parseCount "four")) , checkEqual "reject a word as a number" (Just 0) (parseCount "four") ] main :: IO () main = do forM_ (zip [1 :: Int ..] checks) $ \(number, result) -> if resultPassed result then printf "ok %d - %s\n" number (resultName result) else printf "not ok %d - %s: %s\n" number (resultName result) (resultDetail result) let failed = length (filter (not . resultPassed) checks) printf "%d checks, %d failed\n" (length checks) failed if failed == 0 then exitSuccess else exitFailure

ghc -Wall -O1 -o checks-fail ChecksFail.hs
./checks-fail; echo "status $?"

[1 of 2] Compiling Main ( ChecksFail.hs, ChecksFail.o )
[2 of 2] Linking checks-fail
not ok 1 - clamp below the range: expected 0, got 1
ok 2 - clamp inside the range
ok 3 - reject a word
not ok 4 - reject a word as a number: expected Just 0, got Nothing
4 checks, 2 failed
status 1

The message names both values, so the reader does not have to open the file to see what was compared, and the two failing checks are on one screen. The status is 1, which is the only thing a build server needs: the same command that passed above now fails the build.

A driver that stopped at the first failure would have shown one line and hidden the second mistake, and a driver that always exits 0 would have shown both lines and failed the build nowhere. Those two lines of the driver, length (filter (not . resultPassed) checks) and the if that follows it, are the whole reason the file is worth writing.

What comes next

This driver is the shape every framework keeps. The page on cabal test puts the same checks into a test-suite stanza, where Cabal builds and runs the program and reports PASS or FAIL from its exit status. The property page replaces the fixed inputs with generated ones, and the coverage page measures which lines the checks reached.

The course lesson behind the exit status is arguments and exit codes, and the capstone lesson Capstone A: testing the parser uses a table of good and bad lines with a runner of this shape.

The code on this page was built with GHC 9.10.3 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev