Running the test suite

A suite is a program

An exitcode-stdio-1.0 test suite is a program with a main, and Cabal learns the verdict from its exit status. There is no assertion library in the boot libraries, so the checks of the previous lesson are written by hand:

check :: String -> Bool -> IO Bool check label ok = do putStrLn ((if ok then "ok " else "FAIL ") ++ label) pure ok

main runs the checks, prints a line each and exits with 0 or 1. Everything a framework would add on top is a better report: the interface Cabal cares about is the status.

A check that fails

One expectation is wrong on purpose: the third check expects "0.5" where the formatter gives "0.05". The rest of the file is the suite of the previous lesson.

module Main (main) where import Shop.Price (formatPence, withTax) import System.Exit (exitFailure, exitSuccess) check :: String -> Bool -> IO Bool check label ok = do putStrLn ((if ok then "ok " else "FAIL ") ++ label) pure ok main :: IO () main = do results <- sequence [ check "100 with tax" (withTax 100 == 120) , check "format 120" (formatPence 120 == "1.20") , check "format 5" (formatPence 5 == "0.5") ] if and results then exitSuccess else exitFailure

The suite still compiles and still runs all three checks: a failing expectation is a False in a list, not an exception. That is deliberate, because a run that stopped at the first failure would hide the other two results.

What cabal test prints

cabal test --offline

Build profile: -w ghc-9.10.3 -O1
In order, the following will be built (use -v for more details):
 - shop-0.1.0.0 (test:shop-test) (first run)
Preprocessing test suite 'shop-test' for shop-0.1.0.0...
Building test suite 'shop-test' for shop-0.1.0.0...
Running 1 test suites...
Test suite shop-test: RUNNING...
ok 100 with tax
ok format 120
FAIL format 5
Test suite shop-test: FAIL
0 of 1 test suites (0 of 1 test cases) passed.
Error: [Cabal-7125]
Tests failed for test:shop-test from shop-0.1.0.0.

Four things to read in that output:

As in the previous lesson, Cabal prints the absolute path of the log file between the FAIL line and the count; that line is left out here because it depends on the machine.

The exit status and Cabal

cabal test itself leaves a non-zero status when a suite fails, which is the one thing a CI job checks:

cabal test --offline
echo "status: $?"

status: 1

The status is 1 and not 3 or 64: Cabal reports the failure of the command, and the suite's own status is in the log. That is why a suite should use 0 and 1 and leave the fine-grained codes to the program it tests, as the lesson on arguments did.

The fix and a second run

The expectation becomes "0.05", and the same command recompiles the suite and passes:

cabal test --offline

Build profile: -w ghc-9.10.3 -O1
Running 1 test suites...
Test suite shop-test: RUNNING...
ok 100 with tax
ok format 120
ok format 5
Test suite shop-test: PASS
1 of 1 test suites (1 of 1 test cases) passed.

Only the suite was rebuilt, because only its source changed; the library and the executable were up to date. A second run with no change at all prints PASS without compiling, which makes the test cheap enough to run after every edit. The next lesson leaves the compiler and goes through its flags, including the two that turn a warning into a failure.

Exercises

  1. Break a different check and count how many lines of the suite still appear.
  2. Make check print the expected and the actual value on a failure.
  3. Write a suite that exits with 1 when the list of results is empty and explain why that is a failure.
  4. Run cabal test --offline; echo $? in both states and compare the numbers.
  5. Find the log file under dist-newstyle/ and read the failing run.
  6. Explain why a suite should run every check instead of stopping at the first failure.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. How does Cabal know that a test suite passed?

    Answer

    From the exit status of the suite program

    exitcode-stdio-1.0 means the status is the verdict; the output is for a human.

  2. What does the suite print for a failed check in this lesson?

    Answer

    The word FAIL and the label

    The check function prints FAIL and the label and returns False, so the run continues.

  3. What status does cabal test leave when a suite fails?

    Answer

    1

    Cabal reports the failed command with status 1, whatever the suite used.

  4. What does this program print? module Main (main) where import System.Exit (exitFailure) main :: IO () main = do putStrLn "checking" exitFailure

    Answer

    checking

    The line is printed and the program then exits with status 1.

  5. What does 0 of 1 test suites (0 of 1 test cases) passed count?

    Answer

    Suites, not the individual checks inside them

    Cabal counts suites; the individual checks are the suite output.

  6. Why does the suite run all three checks when the third fails?

    Answer

    The checks are values in a list and the failure is a False, not an exception

    A False in the list does not end the program, so every check runs and the summary decides.

  7. What does this program print? module Main (main) where check :: Bool -> String check ok = if ok then "ok" else "FAIL" main :: IO () main = putStrLn (check (1 + 1 == 3))

    Answer

    FAIL

    The comparison is False, so check returns the failure word.

  8. What is Error: [Cabal-7125] in the failing output?

    Answer

    A Cabal error code for a failed test, which a script can match

    The bracketed code identifies the Cabal error class, like the GHC codes on the compile errors.

  9. What does the second cabal test run compile after only the suite changed?

    Answer

    Only the test suite

    The library and executable are up to date, so only the changed component is rebuilt.

  10. Why is a failing check in the list better than a crash?

    Answer

    The other checks still run, so one run reports every failure

    Collecting booleans lets one run report all the failures, which is what a test run should do.

The code on this page was built with GHC 9.10.3 and cabal-install 3.16.0.0, and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev