Coverage with hpc
A passing suite says that the checks it ran passed. It says nothing about the code they never called: a function with no test looks exactly like a function with ten. Coverage answers the other question, which lines did the run reach, and GHC has the answer built in.
-fhpc compiles the program with a counter at every expression and guard, and the program writes the counters into a .tix file when it exits. The hpc program reads that file together with the .mix files the compiler wrote, and prints percentages. Both programs ship with GHC, so this page needs nothing that is not in the image.
What hpc counts
Four numbers per module are worth reading, and only one of them is the headline. Expressions used is the fraction of the instrumented expressions the run evaluated, and it is the number people quote. Alternatives used is the fraction of the constructors or guards a pattern match took, so a case with three branches that only ever sees two reports 66%. Boolean coverage is the guards and conditions, which is where an untested otherwise shows up. Top-level declarations used is the function nobody called.
A line can be covered and still be wrong, and a program can be 100% covered and wrong. Coverage measures the tests against the code, not the code against the specification, and the next page is the one about what a test asserts.
Two modules and a build
The program has two modules on purpose: coverage is reported per module, and a single file hides which half of the program the run never touched. Report holds the logic, three guards in bucket and a list comprehension in rows, and Main reads the arguments and prints.
module Report (bucket, rows) where
bucket :: Int -> String
bucket amount
| amount < 10 = "small"
| amount < 100 = "medium"
| otherwise = "large"
rows :: [Int] -> [(String, Int)]
rows amounts = [(bucket amount, amount) | amount <- amounts]
The main module imports the other one, so this file cannot be built on its own: the build command names both sources and GHC compiles them in dependency order.
module Main (main) where
import Report (rows)
import System.Environment (getArgs)
main :: IO ()
main = do
args <- getArgs
mapM_ print (rows (map read args))
Two flags keep the build folder clean. -outputdir build puts the .o and .hi files there instead of next to the sources, and -hpcdir hpc puts the .mix files there. -O0 is deliberate: the optimizer removes code that is never called, and the report should describe the code that was written.
ghc -fhpc -hpcdir hpc -outputdir build -O0 -o app Main.hs Report.hs
[1 of 3] Compiling Report ( Report.hs, build/Report.o )
[2 of 3] Compiling Main ( Main.hs, build/Main.o )
[3 of 3] Linking app
The run and the report
The run passes three amounts: 3, 42 and 7. All of them are below a hundred, so the large branch of bucket is never taken, and the report says so.
./app 3 42 7; echo "status $?"
("small",3)
("medium",42)
("small",7)
status 0
The .tix file is written when the program exits, in the folder the program was started from. hpc report reads it and needs the .mix files of the same build; --per-module prints a section for each one.
hpc report app.tix --per-module --hpcdir=hpc
-----<module Main>-----
100% expressions used (8/8)
100% boolean coverage (0/0)
100% guards (0/0)
100% 'if' conditions (0/0)
100% qualifiers (0/0)
100% alternatives used (0/0)
100% local declarations used (0/0)
100% top-level declarations used (1/1)
-----<module Report>-----
87% expressions used (14/16)
33% boolean coverage (1/3)
33% guards (1/3), 1 always True, 1 unevaluated
100% 'if' conditions (0/0)
100% qualifiers (0/0)
66% alternatives used (2/3)
100% local declarations used (0/0)
100% top-level declarations used (2/2)
status 0
Main is at 100% because the run called everything in it. Report is at 87% of expressions and 66% of alternatives: one of the three guards was never evaluated, and the one the run never reached is the large branch, because every argument was below a hundred. The line 33% guards (1/3), 1 always True, 1 unevaluated is hpc being precise about it: the otherwise guard counts as always true and the third one is the guard the run missed.
The .mix files and -hpcdir
Without -hpcdir the compiler puts the .mix files in a folder named .hpc next to the sources, and hpc report looks there by default. Because the build above put them in hpc/, the report without the flag fails, and the first line is the one that matters:
hpc report app.tix; echo "status $?"
hpc-ghc-9.10.3: can not find Main in ./.hpc
CallStack (from HasCallStack):
error, called at libraries/hpc/Trace/Hpc/Mix.hs:110:15 in hpc-0.7.0.2-inplace:Trace.Hpc.Mix
HasCallStack backtrace:
collectBacktraces, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:169:13 in ghc-internal:GHC.Internal.Exception
toExceptionWithBacktrace, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:204:5 in ghc-internal:GHC.Internal.Exception
error, called at libraries/hpc/Trace/Hpc/Mix.hs:110:15 in hpc-0.7.0.2-inplace:Trace.Hpc.Mix
status 1
can not find Main in ./.hpc is the whole problem: the default search path is ./.hpc and the files are somewhere else. The CallStack lines below it are hpc reporting the failure of its own internal call, not a second mistake. The fix is the flag, or a build that leaves everything in the default place.
A run that covers the branches
The .tix file accumulates: a second run adds its counts to the first, which is how a suite of several runs is measured as one. To measure a single run, remove the file first. This run also passes 150 and 999, so the large branch is taken:
rm -f app.tix
./app 3 42 7 150 999 > /dev/null
hpc report app.tix --per-module --hpcdir=hpc
-----<module Main>-----
100% expressions used (8/8)
100% boolean coverage (0/0)
100% guards (0/0)
100% 'if' conditions (0/0)
100% qualifiers (0/0)
100% alternatives used (0/0)
100% local declarations used (0/0)
100% top-level declarations used (1/1)
-----<module Report>-----
100% expressions used (16/16)
66% boolean coverage (2/3)
66% guards (2/3), 1 always True
100% 'if' conditions (0/0)
100% qualifiers (0/0)
100% alternatives used (3/3)
100% local declarations used (0/0)
100% top-level declarations used (2/2)
Every expression is covered now and every alternative. The guards stay at 66%: otherwise is always true, so hpc counts it as a third guard that can never fail on its own, and nothing a program does will change that. A number below 100 that no test can raise is the one thing to know before a project sets a coverage threshold.
The annotated source
hpc markup writes a copy of every source file with the covered lines coloured and the missed ones marked, plus an index that links them. It is the report to look at when a percentage is suspicious:
hpc markup app.tix --hpcdir=hpc --destdir=html
Writing: Main.hs.html
Writing: Report.hs.html
Writing: hpc_index.html
Writing: hpc_index_fun.html
Writing: hpc_index_alt.html
Writing: hpc_index_exp.html
The folder holds one page per module and three indexes, by declaration, by alternative and by expression. The same information is in hpc report; the markup is what a colleague reads without running anything.
What coverage does not say
A line that ran is not a line whose result was checked. The run above evaluated the small branch with a value it printed, and if the expected value in a test were wrong, the coverage would still be 100%. Coverage answers "did the tests reach this", not "did they notice when it broke", and the two questions need the two kinds of test this group covers: assertions for the values and coverage for the reach.
Instrumentation also costs. Every expression gets a counter, so an -fhpc build is slower than an -O0 build and much slower than -O2, which is why coverage is a separate build and never the build that ships. The course lesson on GHC flags covers -O0 to -O2 and the lesson on measuring performance covers what to do with the numbers.
The code on this page was built and measured with GHC 9.10.3 and the hpc that ships with it, and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev