A reusable assert module

The course builds a test module in two lessons: the first assertions and tolerances and exit codes. This page is the version a project keeps: four assertions that cover the values a numerical program compares, a suite written as separate test procedures instead of a run of calls in the main program, a table that dispatches them, and the output a runner reads. It also records one thing this compiler refuses, and the shape that works instead.

The module

The module holds two counters that no procedure outside it can touch, four public assertions and one report. Every assertion writes exactly one line and increments the same counters, so a new assertion cannot forget to count, and the report is the only place that decides what a failure means.

The detail of a failure is built with an internal write into a fixed-length buffer and printed with TRIM, which is the pattern of lesson 048. The buffer is eighty characters, which is enough for two numbers and a word; a project that prints long strings would pass the detail as an argument instead of formatting it inside the assertion.

Four assertions

ASSERT_TRUE takes a condition the caller computed, so a test can check anything. ASSERT_EQUAL_INT prints both values, because an integer comparison that fails is nearly always an off-by-one and the two numbers say which way. ASSERT_EQUAL_STR compares characters with ==, which pads the shorter operand with blanks: 'north' and 'north ' are equal, and a test that wants to catch trailing blanks compares LEN_TRIM as well.

ASSERT_CLOSE takes the tolerance as an argument, which is the decision of the next page in this group. It compares abs(expected-actual) / max(abs(expected),abs(actual)) with the tolerance, so the same number means the same thing at any magnitude, and two zeros are equal without a division.

A suite of test procedures

Three test procedures, each a subroutine that prints its own heading and calls assertions. Writing them as procedures rather than as a sequence of calls in the main program is what makes a suite grow: a procedure has a name, it can be called alone while it is being written, and it can be replaced by another one without touching the others. The functions they test are in the same module, so a test calls them the way a user would.

The dispatch table

The program runs the three procedures in a loop over a table. The table is an array of a derived type whose component is a procedure pointer with the shape of a test:

abstract interface subroutine test_iface() end subroutine test_iface end interface type :: test_case character(len=16) :: name = '' procedure(test_iface), pointer, nopass :: run => null() end type test_case

The type is necessary. gfortran 15.2.0 refuses a plain array of procedure pointers, in every spelling tried here: procedure(test_iface), pointer :: tests(3) gives Error: Syntax error in PROCEDURE statement, and the DIMENSION(3) spelling gives EXTERNAL attribute conflicts with DIMENSION attribute. An array of a type that holds a procedure pointer component is accepted and does the same job, which is the form above and the one the program uses.

Each entry is filled with a name and a target, and the loop calls tests(i)%run. The names are not printed by the table: the test procedures print their own headings, because a heading that a procedure prints itself cannot drift from the procedure that runs.

The passing run

The file asserts.f90 holds all three units: the assertion module, the suite and the program.

module asserts use, intrinsic :: iso_fortran_env, only: dp => real64, output_unit implicit none private public :: assert_true, assert_equal_int, assert_close, assert_equal_str, report integer :: checks = 0 integer :: failures = 0 contains subroutine record(name, ok, detail) character(len=*), intent(in) :: name, detail logical, intent(in) :: ok checks = checks + 1 if (ok) then print '(a,i0,a,a)', 'ok ', checks, ' - ', name else failures = failures + 1 print '(a,i0,a,a,a,a)', 'not ok ', checks, ' - ', name, ': ', trim(detail) end if end subroutine record subroutine assert_true(cond, name) logical, intent(in) :: cond character(len=*), intent(in) :: name call record(name, cond, 'condition is false') end subroutine assert_true subroutine assert_equal_int(expected, actual, name) integer, intent(in) :: expected, actual character(len=*), intent(in) :: name character(len=80) :: detail write (detail, '(a,i0,a,i0)') 'expected ', expected, ', got ', actual call record(name, expected == actual, detail) end subroutine assert_equal_int subroutine assert_close(expected, actual, rtol, name) real(dp), intent(in) :: expected, actual, rtol character(len=*), intent(in) :: name character(len=80) :: detail real(dp) :: scale, err scale = max(abs(expected), abs(actual)) if (scale > 0.0_dp) then err = abs(expected - actual) / scale else err = 0.0_dp end if write (detail, '(a,es10.3,a,es10.3)') 'relative error ', err, ' above tolerance ', rtol call record(name, err <= rtol, detail) end subroutine assert_close subroutine assert_equal_str(expected, actual, name) character(len=*), intent(in) :: expected, actual, name character(len=80) :: detail write (detail, '(a,a,a,a)') 'expected "', trim(expected), '", got "', trim(actual) // '"' call record(name, expected == actual, detail) end subroutine assert_equal_str subroutine report() print '(a,i0,a,i0,a)', '# ', checks, ' checks, ', failures, ' failures' flush (output_unit) if (failures > 0) error stop 1 end subroutine report end module asserts module suite use asserts use, intrinsic :: iso_fortran_env, only: dp => real64 implicit none private public :: test_mean, test_spread, test_names, test_case abstract interface subroutine test_iface() end subroutine test_iface end interface type :: test_case character(len=16) :: name = '' procedure(test_iface), pointer, nopass :: run => null() end type test_case contains subroutine test_mean() print '(a)', '# test_mean' call assert_close(2.0_dp, mean([1.0_dp, 2.0_dp, 3.0_dp]), 1.0e-12_dp, 'mean of 1 2 3') call assert_close(2.0_dp, mean([2.0_dp, 2.0_dp]), 1.0e-12_dp, 'mean of 2 2') call assert_close(0.0_dp, mean([-1.0_dp, 1.0_dp]), 1.0e-12_dp, 'mean of -1 1') end subroutine test_mean subroutine test_spread() print '(a)', '# test_spread' call assert_close(2.0_dp, spread_of([1.0_dp, 2.0_dp, 3.0_dp]), 1.0e-12_dp, 'spread of 1 2 3') call assert_true(abs(spread_of([5.0_dp])) <= 0.0_dp, 'spread of one value is zero') call assert_equal_int(3, size([1.0_dp, 2.0_dp, 3.0_dp]), 'size of a three-element array') end subroutine test_spread subroutine test_names() print '(a)', '# test_names' call assert_equal_str('north', region_name(1), 'region 1 is north') call assert_equal_str('south', region_name(2), 'region 2 is south') call assert_true(len_trim(region_name(3)) == 4, 'region 3 has four letters') end subroutine test_names pure real(dp) function mean(v) result(m) real(dp), intent(in) :: v(:) m = sum(v) / real(size(v), dp) end function mean pure real(dp) function spread_of(v) result(s) real(dp), intent(in) :: v(:) s = maxval(v) - minval(v) end function spread_of pure function region_name(which) result(name) integer, intent(in) :: which character(len=8) :: name select case (which) case (1) name = 'north' case (2) name = 'south' case default name = 'east' end select end function region_name end module suite program run_tests use, intrinsic :: iso_fortran_env, only: dp => real64 use asserts use suite implicit none type(test_case) :: tests(3) character(len=8) :: mode integer :: i tests(1)%name = 'test_mean' tests(1)%run => test_mean tests(2)%name = 'test_spread' tests(2)%run => test_spread tests(3)%name = 'test_names' tests(3)%run => test_names call get_command_argument(1, mode) do i = 1, size(tests) call tests(i)%run end do if (trim(mode) == 'bad') then call assert_equal_int(5, 4, 'four is five, a wrong expectation') end if call report() end program run_tests

gfortran -Wall -Wextra -std=f2018 asserts.f90 -o run_tests
./run_tests

# test_mean
ok 1 - mean of 1 2 3
ok 2 - mean of 2 2
ok 3 - mean of -1 1
# test_spread
ok 4 - spread of 1 2 3
ok 5 - spread of one value is zero
ok 6 - size of a three-element array
# test_names
ok 7 - region 1 is north
ok 8 - region 2 is south
ok 9 - region 3 has four letters
# 9 checks, 0 failures

Nine lines in the Test Anything Protocol form and a status of 0. The three headings group the checks by procedure, which a real runner ignores and a reader does not.

The failing run

A suite that has never failed has not been shown to be able to fail. The program takes an argument that adds one wrong expectation after the real tests, which is a negative control: it must produce a not ok line and a status of 1, and if it ever produces a 0 the harness is broken.

GFORTRAN_ERROR_BACKTRACE=0 ./run_tests bad > bad.txt 2>&1
echo $?
tail -4 bad.txt

1
not ok 10 - four is five, a wrong expectation: expected 5, got 4
# 10 checks, 1 failures
ERROR STOP 1

The redirection puts both streams in one file, which is why the ERROR STOP line comes after the summary: FLUSH (OUTPUT_UNIT) in the report runs before the stop, and the error stream is written after it. In a pipe the two streams have no order between them, which is the trap of the lesson-119 page.

What a project adds

Three things turn this page into a test suite. One program per module under test, each with its own suite, so a failure names a file. A runner that calls the programs, keeps a failure and returns it, in the shape lesson 119 writes. And a build target that compiles them with the checked build of the next pages in this group, because a test run on an unchecked build can pass for the wrong reason.

What the module deliberately does not do, and a framework would: discover tests from a specification, run each in its own process, catch a crash as a failed test rather than a dead runner, and write a report in a machine format. Those are the features the last page of this group describes and does not run.

Exercises

  1. Add an assertion for a real that must be exactly zero. Which descriptor would print the value in the failure, and why is an exact comparison wrong for a computed zero?
  2. Add a fourth test procedure to the suite and to the table. How many places change, and what happens if you forget the name?
  3. Make the suite run the tests from the command line, one by number. Which part of the table do you need then?
  4. Run the suite under valgrind --error-exitcode=1. What does the status become, and what does that tell you about a runner that only reads it?

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev

Banner image