A reusable assert module
The course builds a test module in two lessons: the first assertions and tolerances and exit codes. This page is the version a project keeps: four assertions that cover the values a numerical program compares, a suite written as separate test procedures instead of a run of calls in the main program, a table that dispatches them, and the output a runner reads. It also records one thing this compiler refuses, and the shape that works instead.
The module
The module holds two counters that no procedure outside it can touch, four public assertions and one report. Every assertion writes exactly one line and increments the same counters, so a new assertion cannot forget to count, and the report is the only place that decides what a failure means.
The detail of a failure is built with an internal write into a fixed-length buffer and printed with TRIM, which is the pattern of lesson 048. The buffer is eighty characters, which is enough for two numbers and a word; a project that prints long strings would pass the detail as an argument instead of formatting it inside the assertion.
Four assertions
ASSERT_TRUE takes a condition the caller computed, so a test can check anything. ASSERT_EQUAL_INT prints both values, because an integer comparison that fails is nearly always an off-by-one and the two numbers say which way. ASSERT_EQUAL_STR compares characters with ==, which pads the shorter operand with blanks: 'north' and 'north ' are equal, and a test that wants to catch trailing blanks compares LEN_TRIM as well.
ASSERT_CLOSE takes the tolerance as an argument, which is the decision of the next page in this group. It compares abs(expected-actual) / max(abs(expected),abs(actual)) with the tolerance, so the same number means the same thing at any magnitude, and two zeros are equal without a division.
A suite of test procedures
Three test procedures, each a subroutine that prints its own heading and calls assertions. Writing them as procedures rather than as a sequence of calls in the main program is what makes a suite grow: a procedure has a name, it can be called alone while it is being written, and it can be replaced by another one without touching the others. The functions they test are in the same module, so a test calls them the way a user would.
The dispatch table
The program runs the three procedures in a loop over a table. The table is an array of a derived type whose component is a procedure pointer with the shape of a test:
abstract interface
subroutine test_iface()
end subroutine test_iface
end interface
type :: test_case
character(len=16) :: name = ''
procedure(test_iface), pointer, nopass :: run => null()
end type test_case
The type is necessary. gfortran 15.2.0 refuses a plain array of procedure pointers, in every spelling tried here: procedure(test_iface), pointer :: tests(3) gives Error: Syntax error in PROCEDURE statement, and the DIMENSION(3) spelling gives EXTERNAL attribute conflicts with DIMENSION attribute. An array of a type that holds a procedure pointer component is accepted and does the same job, which is the form above and the one the program uses.
Each entry is filled with a name and a target, and the loop calls tests(i)%run. The names are not printed by the table: the test procedures print their own headings, because a heading that a procedure prints itself cannot drift from the procedure that runs.
The passing run
The file asserts.f90 holds all three units: the assertion module, the suite and the program.
module asserts
use, intrinsic :: iso_fortran_env, only: dp => real64, output_unit
implicit none
private
public :: assert_true, assert_equal_int, assert_close, assert_equal_str, report
integer :: checks = 0
integer :: failures = 0
contains
subroutine record(name, ok, detail)
character(len=*), intent(in) :: name, detail
logical, intent(in) :: ok
checks = checks + 1
if (ok) then
print '(a,i0,a,a)', 'ok ', checks, ' - ', name
else
failures = failures + 1
print '(a,i0,a,a,a,a)', 'not ok ', checks, ' - ', name, ': ', trim(detail)
end if
end subroutine record
subroutine assert_true(cond, name)
logical, intent(in) :: cond
character(len=*), intent(in) :: name
call record(name, cond, 'condition is false')
end subroutine assert_true
subroutine assert_equal_int(expected, actual, name)
integer, intent(in) :: expected, actual
character(len=*), intent(in) :: name
character(len=80) :: detail
write (detail, '(a,i0,a,i0)') 'expected ', expected, ', got ', actual
call record(name, expected == actual, detail)
end subroutine assert_equal_int
subroutine assert_close(expected, actual, rtol, name)
real(dp), intent(in) :: expected, actual, rtol
character(len=*), intent(in) :: name
character(len=80) :: detail
real(dp) :: scale, err
scale = max(abs(expected), abs(actual))
if (scale > 0.0_dp) then
err = abs(expected - actual) / scale
else
err = 0.0_dp
end if
write (detail, '(a,es10.3,a,es10.3)') 'relative error ', err, ' above tolerance ', rtol
call record(name, err <= rtol, detail)
end subroutine assert_close
subroutine assert_equal_str(expected, actual, name)
character(len=*), intent(in) :: expected, actual, name
character(len=80) :: detail
write (detail, '(a,a,a,a)') 'expected "', trim(expected), '", got "', trim(actual) // '"'
call record(name, expected == actual, detail)
end subroutine assert_equal_str
subroutine report()
print '(a,i0,a,i0,a)', '# ', checks, ' checks, ', failures, ' failures'
flush (output_unit)
if (failures > 0) error stop 1
end subroutine report
end module asserts
module suite
use asserts
use, intrinsic :: iso_fortran_env, only: dp => real64
implicit none
private
public :: test_mean, test_spread, test_names, test_case
abstract interface
subroutine test_iface()
end subroutine test_iface
end interface
type :: test_case
character(len=16) :: name = ''
procedure(test_iface), pointer, nopass :: run => null()
end type test_case
contains
subroutine test_mean()
print '(a)', '# test_mean'
call assert_close(2.0_dp, mean([1.0_dp, 2.0_dp, 3.0_dp]), 1.0e-12_dp, 'mean of 1 2 3')
call assert_close(2.0_dp, mean([2.0_dp, 2.0_dp]), 1.0e-12_dp, 'mean of 2 2')
call assert_close(0.0_dp, mean([-1.0_dp, 1.0_dp]), 1.0e-12_dp, 'mean of -1 1')
end subroutine test_mean
subroutine test_spread()
print '(a)', '# test_spread'
call assert_close(2.0_dp, spread_of([1.0_dp, 2.0_dp, 3.0_dp]), 1.0e-12_dp, 'spread of 1 2 3')
call assert_true(abs(spread_of([5.0_dp])) <= 0.0_dp, 'spread of one value is zero')
call assert_equal_int(3, size([1.0_dp, 2.0_dp, 3.0_dp]), 'size of a three-element array')
end subroutine test_spread
subroutine test_names()
print '(a)', '# test_names'
call assert_equal_str('north', region_name(1), 'region 1 is north')
call assert_equal_str('south', region_name(2), 'region 2 is south')
call assert_true(len_trim(region_name(3)) == 4, 'region 3 has four letters')
end subroutine test_names
pure real(dp) function mean(v) result(m)
real(dp), intent(in) :: v(:)
m = sum(v) / real(size(v), dp)
end function mean
pure real(dp) function spread_of(v) result(s)
real(dp), intent(in) :: v(:)
s = maxval(v) - minval(v)
end function spread_of
pure function region_name(which) result(name)
integer, intent(in) :: which
character(len=8) :: name
select case (which)
case (1)
name = 'north'
case (2)
name = 'south'
case default
name = 'east'
end select
end function region_name
end module suite
program run_tests
use, intrinsic :: iso_fortran_env, only: dp => real64
use asserts
use suite
implicit none
type(test_case) :: tests(3)
character(len=8) :: mode
integer :: i
tests(1)%name = 'test_mean'
tests(1)%run => test_mean
tests(2)%name = 'test_spread'
tests(2)%run => test_spread
tests(3)%name = 'test_names'
tests(3)%run => test_names
call get_command_argument(1, mode)
do i = 1, size(tests)
call tests(i)%run
end do
if (trim(mode) == 'bad') then
call assert_equal_int(5, 4, 'four is five, a wrong expectation')
end if
call report()
end program run_tests
gfortran -Wall -Wextra -std=f2018 asserts.f90 -o run_tests
./run_tests
# test_mean
ok 1 - mean of 1 2 3
ok 2 - mean of 2 2
ok 3 - mean of -1 1
# test_spread
ok 4 - spread of 1 2 3
ok 5 - spread of one value is zero
ok 6 - size of a three-element array
# test_names
ok 7 - region 1 is north
ok 8 - region 2 is south
ok 9 - region 3 has four letters
# 9 checks, 0 failures
Nine lines in the Test Anything Protocol form and a status of 0. The three headings group the checks by procedure, which a real runner ignores and a reader does not.
The failing run
A suite that has never failed has not been shown to be able to fail. The program takes an argument that adds one wrong expectation after the real tests, which is a negative control: it must produce a not ok line and a status of 1, and if it ever produces a 0 the harness is broken.
GFORTRAN_ERROR_BACKTRACE=0 ./run_tests bad > bad.txt 2>&1
echo $?
tail -4 bad.txt
1
not ok 10 - four is five, a wrong expectation: expected 5, got 4
# 10 checks, 1 failures
ERROR STOP 1
The redirection puts both streams in one file, which is why the ERROR STOP line comes after the summary: FLUSH (OUTPUT_UNIT) in the report runs before the stop, and the error stream is written after it. In a pipe the two streams have no order between them, which is the trap of the lesson-119 page.
What a project adds
Three things turn this page into a test suite. One program per module under test, each with its own suite, so a failure names a file. A runner that calls the programs, keeps a failure and returns it, in the shape lesson 119 writes. And a build target that compiles them with the checked build of the next pages in this group, because a test run on an unchecked build can pass for the wrong reason.
What the module deliberately does not do, and a framework would: discover tests from a specification, run each in its own process, catch a crash as a failed test rather than a dead runner, and write a report in a machine format. Those are the features the last page of this group describes and does not run.
Exercises
- Add an assertion for a real that must be exactly zero. Which descriptor would print the value in the failure, and why is an exact comparison wrong for a computed zero?
- Add a fourth test procedure to the suite and to the table. How many places change, and what happens if you forget the name?
- Make the suite run the tests from the command line, one by number. Which part of the table do you need then?
- Run the suite under valgrind --error-exitcode=1. What does the status become, and what does that tell you about a runner that only reads it?
The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev