Testing: drivers, tolerances and exit codes
A test that compares two reals needs a tolerance, and the tolerance is the part that decides whether the suite is useful. Too loose and a wrong answer passes; too tight and a correct answer fails on the last bit. This lesson builds a driver that checks a Taylor series for exp(1) against the intrinsic, watches the error fall with the number of terms, hits the floor of single precision, and reaches the floor of double precision with the same code written for another kind. Then a shell script runs the driver twice and turns the statuses into one answer.
A table of cases
The module is the one from the previous lesson with one change: the tolerance is relative instead of absolute. The error is abs(expected - actual) / max(abs(expected), abs(actual)), so the same tolerance means the same thing for a value near zero and a value near a million. A relative check on two zeros is the one case the formula cannot express, and the else branch compares them absolutely.
The driver holds the cases in code rather than in a table, because each case has a name and a tolerance as well as values, and a derived type with a character component would be more machinery than the three calls need. The strict argument adds the two cases that are meant to fail: a single-precision series against a tolerance below its precision, and the same series in double precision against the same tolerance.
The file test_taylor.f90, the module and the program in one file:
module tolerance
use, intrinsic :: iso_fortran_env, only: dp => real64, output_unit
implicit none
private
public :: assert_rel, assert_rel64, report
integer :: checks = 0
integer :: failures = 0
contains
subroutine record(name, error, rtol)
character(len=*), intent(in) :: name
real, intent(in) :: error, rtol
character(len=96) :: detail
checks = checks + 1
write (detail, '(a,es10.3)') 'rel err ', error
if (error <= rtol) then
print '(a,i0,a,a,a,a,a)', 'ok ', checks, ' - ', name, ' (', trim(detail), ')'
else
failures = failures + 1
write (detail, '(a,es10.3,a,es10.3)') 'rel err ', error, ' > rtol ', rtol
print '(a,i0,a,a,a,a)', 'not ok ', checks, ' - ', name, ': ', trim(detail)
end if
end subroutine record
subroutine assert_rel(expected, actual, rtol, name)
real, intent(in) :: expected, actual, rtol
character(len=*), intent(in) :: name
real :: scale
scale = max(abs(expected), abs(actual))
if (scale > 0.0) then
call record(name, abs(expected - actual) / scale, rtol)
else
call record(name, abs(expected - actual), rtol)
end if
end subroutine assert_rel
subroutine assert_rel64(expected, actual, rtol, name)
real(dp), intent(in) :: expected, actual, rtol
character(len=*), intent(in) :: name
real(dp) :: scale
scale = max(abs(expected), abs(actual))
if (scale > 0.0_dp) then
call record(name, real(abs(expected - actual) / scale), real(rtol))
else
call record(name, real(abs(expected - actual)), real(rtol))
end if
end subroutine assert_rel64
subroutine report()
print '(a,i0,a,i0,a)', '# ', checks, ' checks, ', failures, ' failures'
flush (output_unit)
if (failures > 0) error stop 1
end subroutine report
end module tolerance
program test_taylor
use tolerance
use, intrinsic :: iso_fortran_env, only: dp => real64
implicit none
character(len=16) :: mode
call get_command_argument(1, mode)
call assert_rel(exp(1.0), taylor(1.0, 5), 1.0e-3, 'single, 5 terms')
call assert_rel(exp(1.0), taylor(1.0, 10), 1.0e-6, 'single, 10 terms')
if (trim(mode) == 'strict') then
call assert_rel(exp(1.0), taylor(1.0, 20), 1.0e-12, 'single, 20 terms')
call assert_rel64(exp(1.0_dp), taylor64(1.0_dp, 20), 1.0e-12_dp, 'double, 20 terms')
end if
call report()
contains
pure real function taylor(x, n) result(s)
real, intent(in) :: x
integer, intent(in) :: n
real :: term
integer :: k
s = 1.0
term = 1.0
do k = 1, n
term = term * x / real(k)
s = s + term
end do
end function taylor
pure real(dp) function taylor64(x, n) result(s)
real(dp), intent(in) :: x
integer, intent(in) :: n
real(dp) :: term
integer :: k
s = 1.0_dp
term = 1.0_dp
do k = 1, n
term = term * x / real(k, dp)
s = s + term
end do
end function taylor64
end program test_taylor
Absolute and relative tolerance
An absolute tolerance says abs(a - b) <= tol, and it means a different thing at every scale: 1.0e-6 is a strict test for a value near one and a meaningless one for a value near 1.0e9, where a single-precision real cannot even hold six digits. The relative form divides by the larger of the two values, so 1.0e-6 means six correct digits whatever the magnitude.
The driver uses a different tolerance per case on purpose: 1.0e-3 for a five-term series that is only a rough approximation, 1.0e-6 for ten terms, and 1.0e-12 for the two cases of the strict run. A single tolerance for all of them would either pass the first case for the wrong reason or fail the last one for a reason that is not about the code.
The precision sets a floor
The numbers below are the real errors of the four cases, from the runs in the next section:
| case | tolerance | relative error | result |
|---|---|---|---|
| single, 5 terms | 1.0e-3 | 5.941e-04 | ok |
| single, 10 terms | 1.0e-6 | 8.771e-08 | ok |
| single, 20 terms | 1.0e-12 | 8.771e-08 | not ok |
| double, 20 terms | 1.0e-12 | 1.634e-16 | ok |
The third line is the interesting one. Twenty terms of the series would be more accurate than ten in exact arithmetic, and the error does not move at all, because the sum has reached the resolution of a single-precision real: the next term is smaller than the last bit of the running total, so adding it changes nothing. That floor is about 1.2e-7, which is epsilon(1.0), and no tolerance below it can ever pass for a single-precision computation.
The fourth line is the same algorithm written for real64, and it reaches 1.634e-16, which is the double-precision floor of about 2.2e-16. A test suite that wants twelve correct digits asks for a kind that has them; a tolerance alone cannot buy precision.
Two runs, two statuses
gfortran -Wall -Wextra -std=f2018 test_taylor.f90 -o test_taylor
./test_taylor
ok 1 - single, 5 terms (rel err 5.941E-04)
ok 2 - single, 10 terms (rel err 8.771E-08)
# 2 checks, 0 failures
The quick run is the suite that a developer runs after every change, and its status is 0. The strict run adds the two cases that decide whether the tolerance is honest:
GFORTRAN_ERROR_BACKTRACE=0 ./test_taylor strict
echo $?
ok 1 - single, 5 terms (rel err 5.941E-04)
ok 2 - single, 10 terms (rel err 8.771E-08)
not ok 3 - single, 20 terms: rel err 8.771E-08 > rtol 1.000E-12
ok 4 - double, 20 terms (rel err 1.634E-16)
# 4 checks, 1 failures
ERROR STOP 1
1
The failing check does not stop the run: the fourth case still runs and passes, which is what the module's counters are for. The status is 1, and that is the only thing a script needs from the text.
A shell runner
A runner over several test programs is a loop that records a failure and keeps going. The file run-tests.sh:
#!/bin/sh # Run each test program and keep going after a failure. status=0 for mode in quick strict; do if ./test_taylor "$mode" > "log-$mode.txt" 2>&1; then echo "PASS test_taylor $mode" else echo "FAIL test_taylor $mode" status=1 fi done exit "$status"
chmod +x run-tests.sh
GFORTRAN_ERROR_BACKTRACE=0 ./run-tests.sh
echo $?
PASS test_taylor quick
FAIL test_taylor strict
1
One line per test program, and the status of the whole run is the status of the worst case. The log of the failing mode is in log-strict.txt and reads in the right order:
ok 1 - single, 5 terms (rel err 5.941E-04)
ok 2 - single, 10 terms (rel err 8.771E-08)
not ok 3 - single, 20 terms: rel err 8.771E-08 > rtol 1.000E-12
ok 4 - double, 20 terms (rel err 1.634E-16)
# 4 checks, 1 failures
ERROR STOP 1
The flush (output_unit) before ERROR STOP is why. When the program's output goes to a file rather than to a terminal, the C library buffers it, and the runtime writes ERROR STOP to the error stream at once. Without the flush the line appears at the top of the log, before the checks that explain it, and a reader concludes that the summary was lost.
A real project runs one program per module under test, puts the runner in the build as a make check target, and lets the status of the script decide whether the build is good. The test build flags of lesson 114 belong in that target: a suite that runs the checked build finds the out-of-bounds read that made a test pass for the wrong reason.
Rules for a driver
State the tolerance per test, in relative form, and pick it from the precision of the type, not from the first number that passed. Do not stop at the first failure in a suite whose failures tend to be independent. Flush the standard output before ERROR STOP so the log keeps its order. Return the failure in the exit status, because that is the only part a build tool reads. And when a tolerance has to be very tight, ask whether the type can hold it at all: the floor of a single-precision real is 8.771e-08 for this series, and no test below that can pass.
Exercises
- Change the ten-term tolerance to 1.0e-8 and run the quick suite. Which case fails, and what does the message say the error is?
- Add a case for exp(-10.0) with five terms. Compare the relative and the absolute error. Which of the two would a loose absolute tolerance have hidden?
- Make the runner write one line per check instead of one per program. What does it need from the output, and why is the status no longer enough?
- Remove the flush call and run the script again. Where does ERROR STOP 1 appear in the log?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Why is a relative tolerance better than an absolute one?
Answer
It means the same number of correct digits at any magnitude
An absolute tolerance means a different thing at each scale.
-
What relative error does the ten-term single-precision series have?
Answer
8.771e-08
It is the number on this page.
-
Why does the twenty-term single-precision series not improve on ten terms?
Answer
The sum has reached the resolution of a single-precision real
The next term is smaller than the last bit of the total.
-
What is the single-precision floor called?
Answer
epsilon(1.0), about 1.2e-7
The difference between 1 and the next representable value.
-
How does the double-precision case reach 1.634e-16?
Answer
The same algorithm with real64 values
The kind has the digits; the algorithm did not change.
-
What does the driver do after a failing check?
Answer
It records the failure and runs the remaining cases
One run shows every broken expectation.
-
Why flush the output before ERROR STOP?
Answer
Buffered output written to a file would otherwise land after the error line
The order of the log is what a reader needs.
-
What does the shell runner return when one mode fails?
Answer
1, because the status is kept across the loop
The worst status wins.
-
Why does a relative check need a special case for two zeros?
Answer
The division by the maximum is 0/0, so the absolute difference is used
The scale is zero and the formula cannot express it.
-
Which flags belong on the test build of this driver?
Answer
-g -fcheck=all -ffpe-trap=invalid,zero,overflow
Checks and traps catch the bug a tolerance hides.
The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev