Coverage the tests do not give you

A suite that passes proves that the code it ran is right, and says nothing about the code it did not run. Lesson 117 shows how gcov counts what a program executed; this page is about the step after that: a module with three pricing rules and a guard, two tests that pass, and a coverage report that names the rule nobody tested. Then the test that closes the gap, and a script that fails a build when the covered fraction falls.

The module and its tests

The module has one pure function with four paths: a negative amount pays nothing, a regular customer pays the list price, a member pays ten per cent less and staff pay twenty-five per cent less. The test program checks two of them and takes an argument that adds the other two, so the same binary can be the suite before and after the gap is found.

module pricing use, intrinsic :: iso_fortran_env, only: dp => real64 implicit none private public :: discount contains pure function discount(amount, kind) result(d) real(dp), intent(in) :: amount integer, intent(in) :: kind real(dp) :: d if (amount < 0.0_dp) then d = 0.0_dp return end if select case (kind) case (1) d = 0.0_dp case (2) d = 0.10_dp * amount case (3) d = 0.25_dp * amount case default d = 0.0_dp end select end function discount end module pricing program test_pricing use pricing use, intrinsic :: iso_fortran_env, only: dp => real64, output_unit implicit none integer :: checks, failures character(len=8) :: mode call get_command_argument(1, mode) checks = 0 failures = 0 call check_near(0.0_dp, discount(100.0_dp, 1), 'regular customer pays the list price') call check_near(10.0_dp, discount(100.0_dp, 2), 'member gets ten per cent') if (trim(mode) == 'full') then call check_near(25.0_dp, discount(100.0_dp, 3), 'staff gets twenty-five per cent') call check_near(0.0_dp, discount(-5.0_dp, 1), 'a negative amount pays nothing') end if print '(a,i0,a,i0,a)', '# ', checks, ' checks, ', failures, ' failures' flush (output_unit) if (failures > 0) error stop 1 contains subroutine check_near(expected, actual, name) real(dp), intent(in) :: expected, actual character(len=*), intent(in) :: name checks = checks + 1 if (abs(expected - actual) <= 1.0e-12_dp) then print '(a,i0,a,a)', 'ok ', checks, ' - ', name else failures = failures + 1 print '(a,i0,a,a)', 'not ok ', checks, ' - ', name end if end subroutine check_near end program test_pricing

The build is two steps, and the order matters. -c --coverage writes the .gcno file beside the object; the link adds the coverage runtime. Compiling and linking in one command names the coverage files after the executable (tests-pricing.gcno), which is why a project that calls gcov pricing.f90 compiles in two steps.

gfortran -Wall -Wextra -std=f2018 -c --coverage pricing.f90
gfortran --coverage pricing.o -o tests
./tests
gcov -b pricing.f90

ok 1 - regular customer pays the list price
ok 2 - member gets ten per cent
# 2 checks, 0 failures
File 'pricing.f90'
Lines executed:76.67% of 30
Branches executed:100.00% of 14
Taken at least once:50.00% of 14
Calls executed:70.00% of 30
Creating 'pricing.f90.gcov'

The gcov report

The listing puts a count beside every executable line and ##### beside the ones that never ran. The function body is the part worth reading:

function __pricing_MOD_discount called 2 returned 100% blocks executed 67%
        2:    9:  pure function discount(amount, kind) result(d)
        -:   10:    real(dp), intent(in) :: amount
        -:   11:    integer, intent(in) :: kind
        -:   12:    real(dp) :: d
        -:   13:
        2:   14:    if (amount < 0.0_dp) then
branch  0 taken 0% (fallthrough)
branch  1 taken 100%
    #####:   15:      d = 0.0_dp
    #####:   16:      return
        -:   17:    end if
        3:   18:    select case (kind)
        -:   19:    case (1)
        1:   20:      d = 0.0_dp
        -:   21:    case (2)
       1*:   22:      d = 0.10_dp * amount
        -:   23:    case (3)
    #####:   24:      d = 0.25_dp * amount
        -:   25:    case default
       2*:   26:      d = 0.0_dp
branch  0 taken 50%
branch  1 taken 50%
branch  2 taken 0%
branch  3 taken 0%
        -:   27:    end select
        2:   28:  end function discount
        -:   29:
        -:   30:end module pricing

A line marked 1* is a line whose count is one but whose blocks were not all executed: line 22, the member discount, is an assignment that ran, and the star says some of the blocks around it did not. Line 26, the case default arm, is 2* for the same reason and is a warning about the SELECT CASE: the count belongs to the statement, not to the arm.

What the tests never ran

Three things are missing and the report names all three. The guard at lines 15 and 16 never ran, because no test passes a negative amount; the branch line under it says taken 0% for the same reason. The staff rule at line 24 never ran. And the case default arm was reached but never taken, which is what Taken at least once: 50.00% in the summary is counting.

Coverage found this without reading the code, and the fix is a test rather than a change to the module. That is the value of the tool in one sentence: it tells a writer which test is missing, not which line is wrong.

The test that closes the gap

The same binary with the full argument adds the staff rule and the negative amount. The counters of the previous run are still on disk, so a coverage run that is meant to measure one suite deletes the .gcda files first, as lesson 117 explains:

rm -f ./*.gcda
./tests full
gcov -b pricing.f90

ok 1 - regular customer pays the list price
ok 2 - member gets ten per cent
ok 3 - staff gets twenty-five per cent
ok 4 - a negative amount pays nothing
# 4 checks, 0 failures
File 'pricing.f90'
Lines executed:93.33% of 30
Branches executed:100.00% of 14
Taken at least once:64.29% of 14
Calls executed:76.67% of 30
Creating 'pricing.f90.gcov'

Lines went from 76.67 to 93.33 per cent and the taken-branches line from 50 to 64.29. The six and a half per cent that are still missing are the case default arm, which no test takes because no test passes an unknown kind. A suite can reach a hundred per cent, and a project should decide whether it wants to: the default arm of a SELECT CASE is there for the value nobody expected, and a test for it is a test of the fallback.

A coverage gate

A number that nobody checks drifts. The script below runs the suite, reads the percentage out of gcov's summary line and fails when it is below ninety. The file coverage-gate.sh:

#!/bin/sh # Fail when line coverage is below the threshold. rm -f ./*.gcda ./tests full > /dev/null pct=$(gcov pricing.f90 | sed -n 's/^Lines executed:\([0-9.]*\)%.*/\1/p') echo "line coverage: $pct%" awk -v p="$pct" 'BEGIN { exit (p < 90) }' if [ $? -ne 0 ]; then echo "below the 90 per cent gate" >&2 exit 1 fi echo "above the 90 per cent gate"

chmod +x coverage-gate.sh
./coverage-gate.sh
echo $?

line coverage: 93.33
93.33%
above the 90 per cent gate
0

The 93.33% line in the middle is gcov's own trailing summary, which the script does not capture and which lands on the terminal; a project that wants a quiet gate sends it to /dev/null as well. The arithmetic is awk rather than test because the shell cannot compare a decimal, and awk is in the image while bc is not.

What coverage does not say

A hundred per cent of the lines and half of the branches is the usual shape of a suite written from the happy path, and the summary prints the two numbers apart for exactly that reason. Coverage does not say that a value was checked: a test that calls a function and ignores its result covers every line of it. It does not say that the assertions are right, which is the business of the tolerance page, and it does not say that the checked build was used, which is the next page. What it says, precisely, is which lines ran, and that is enough to find the test nobody wrote.

Exercises

  1. Add a test that passes kind = 9 and see the coverage of the default arm. What is the percentage now?
  2. Run the gate with the suite of the first section instead of full. Which line of the script reports the failure, and what is the status?
  3. Add a second module with its own function and compile it in the same run. What does gcov pricing.f90 other.f90 print, and how many summaries?
  4. Compile the tests with -O2 and coverage. How do the line counts change, and which lines does the optimiser merge?

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container, with gcov (Alpine 15.2.0) 15.2.0.

Article author: Arthur Isaev

Banner image