Testing: a hand-written assert module

Fortran has no test framework in its standard library and this image has none installed, so the first test suite of a Fortran project is usually a module of about sixty lines: a counter, three assertions, and a report that stops the program when a check failed. The output follows a convention a runner can read, and the exit status does the rest. This lesson writes that module, uses it to test two functions, and shows a passing and a failing run.

Why a module of your own

A framework buys three things: a way to write an assertion, a way to report, and a way for the build to fail when a test fails. The first two are a subroutine each and the third is ERROR STOP. What a framework adds on top is a runner that discovers test files, temporary directories per test and a report format for a browser; a project that needs those installs one, and a project that needs the first two writes them.

The convention used here is the Test Anything Protocol, which is a line per check. ok 1 - name is a pass, not ok 2 - name: detail is a failure, and the last line is a count. Every runner that speaks TAP can read this output, and so can a reader.

The assert module

The counters are module variables, so every assertion in every procedure of the program adds to the same two numbers. PRIVATE at the top and PUBLIC for the four procedures keeps the counters inside the module: a test program can call the assertions and cannot reset the count behind their back.

The module and the test program are one file here, which is what the checker of this course builds as one program. A real project keeps them in two files and compiles the module first, as lesson 111 shows.

The file test_mean.f90 is below: the module, then the program that uses it.

module asserts implicit none private public :: assert_true, assert_equal_int, assert_close, report integer :: checks = 0 integer :: failures = 0 contains subroutine pass(name) character(len=*), intent(in) :: name checks = checks + 1 print '(a,i0,a,a)', 'ok ', checks, ' - ', name end subroutine pass subroutine fail(name, detail) character(len=*), intent(in) :: name, detail checks = checks + 1 failures = failures + 1 print '(a,i0,a,a,a,a)', 'not ok ', checks, ' - ', name, ': ', detail end subroutine fail subroutine assert_true(cond, name) logical, intent(in) :: cond character(len=*), intent(in) :: name if (cond) then call pass(name) else call fail(name, 'condition is false') end if end subroutine assert_true subroutine assert_equal_int(expected, actual, name) integer, intent(in) :: expected, actual character(len=*), intent(in) :: name character(len=64) :: detail if (expected == actual) then call pass(name) else write (detail, '(a,i0,a,i0)') 'expected ', expected, ', got ', actual call fail(name, trim(detail)) end if end subroutine assert_equal_int subroutine assert_close(expected, actual, tol, name) real, intent(in) :: expected, actual, tol character(len=*), intent(in) :: name character(len=64) :: detail if (abs(expected - actual) <= tol) then call pass(name) else write (detail, '(a,es12.4,a,es12.4)') 'expected ', expected, ', got ', actual call fail(name, trim(detail)) end if end subroutine assert_close subroutine report() print '(a,i0,a,i0,a)', '# ', checks, ' checks, ', failures, ' failures' if (failures > 0) error stop 1 end subroutine report end module asserts program test_mean use asserts implicit none real :: x(3) character(len=32) :: mode call get_command_argument(1, mode) x = [1.0, 2.0, 3.0] call assert_close(2.0, mean(x), 1.0e-6, 'mean of 1 2 3') call assert_close(2.0, mean([2.0, 2.0]), 1.0e-6, 'mean of 2 2') call assert_true(mean([5.0]) > 0.0, 'mean of one positive value') call assert_equal_int(3, size(x), 'size of x') call assert_close(2.0, spread_of(x), 1.0e-6, 'spread of 1 2 3') if (trim(mode) == 'bad') then call assert_close(1.0, mean(x), 1.0e-6, 'mean is 1, a wrong expectation') end if call report() contains pure real function mean(v) result(m) real, intent(in) :: v(:) m = sum(v) / size(v) end function mean pure real function spread_of(v) result(s) real, intent(in) :: v(:) s = maxval(v) - minval(v) end function spread_of end program test_mean

The test program

The second half of the file is the driver. It calls the assertions on two functions of its own, mean and spread_of, and the last call is report. A command-line argument, bad, adds one check with a wrong expectation, which is how the failing run below is produced without a second source file. The tolerance is 1.0e-6 in every real comparison, because single precision carries about seven digits and a sum of three values cannot be exact in general.

The functions live in the CONTAINS section of the program, so they are internal procedures and the driver can call them without an interface. A real project puts them in a module of their own and tests that module, which is the same code with the units the other way round.

A passing run

gfortran -Wall -Wextra -std=f2018 test_mean.f90 -o test_mean
./test_mean

ok 1 - mean of 1 2 3
ok 2 - mean of 2 2
ok 3 - mean of one positive value
ok 4 - size of x
ok 5 - spread of 1 2 3
# 5 checks, 0 failures

Five lines and a count, and the status is 0. A shell script can run the binary and read the status, and a CI job that does nothing but that already fails the build when a check fails.

A failing run

With the wrong expectation the sixth line is a failure, the count says one, and ERROR STOP 1 ends the program with status 1:

GFORTRAN_ERROR_BACKTRACE=0 ./test_mean bad
echo $?

ok 1 - mean of 1 2 3
ok 2 - mean of 2 2
ok 3 - mean of one positive value
ok 4 - size of x
ok 5 - spread of 1 2 3
not ok 6 - mean is 1, a wrong expectation: expected   1.0000E+00, got   2.0000E+00
# 6 checks, 1 failures
ERROR STOP 1
1

The detail after the colon is the part a reader acts on: expected one, got two. The variable is set to zero so that the address list after ERROR STOP does not drown the six lines that matter; without it the runtime prints a backtrace of its own, as lesson 028 shows.

A failing test does not stop at the first failure, which is a decision worth making on purpose. This module records every check and reports at the end, so one run shows every broken expectation; a suite that returns after the first failure is faster to read when the first failure is usually the cause of the rest. Both are defensible, and a project should pick one and stay with it.

The frameworks not in this image

Three projects do this work with more features, and none of them is installed here: pFUnit, which generates test modules from a specification and runs them through a Python driver; test-drive, a small Fortran library whose check is close to the assertions above; and fortran-stdlib, a community standard library of modules for sorting, strings and statistics, each with its own tests. fpm has a fpm test command that builds and runs a project's test programs. This page shows no output of any of them, because none was run: the route is a package manager or a source checkout per project, and then the framework's own test command.

The module above is not a competitor to them. It is what a project starts with, and a project that outgrows it migrates its assertions one at a time, keeping the names and the TAP lines.

Rules for a test suite

Test the small pure functions first: they need no input, no file and no clock, and a failure names one function. Compare reals with a tolerance and write the tolerance in the call. Keep the counters in a module and the report in one place, so a new assertion cannot forget to count. Build the tests with the flags of lesson 114 (-g -fcheck=all -ffpe-trap=invalid,zero,overflow), because a test that reads out of bounds and passes is worse than a test that fails.

And run the suite from a script that stops on a non-zero status, so that the exit code of ERROR STOP reaches the build. The next lesson is about the driver around several test programs, the tolerances that differ per case, and what a runner does with the status.

Exercises

  1. Add an assert_equal_real that compares exactly, without a tolerance. Which test in the driver would fail with it, and why?
  2. Make report return the number of failures as its result instead of stopping. How does the driver then set the exit status?
  3. Add a check for mean([]). What happens, and what should the test assert about the size before it calls mean?
  4. Run the passing program under valgrind --leak-check=full and check ERROR SUMMARY. What does it say about the module's counters?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. Why does a Fortran project often write its own assert module?

    Answer

    The standard library has no test framework and this image has none installed

    Sixty lines cover assertions, reporting and the exit status.

  2. Where do the two counters live?

    Answer

    In the module, so every assertion adds to the same numbers

    Module variables are shared by every user of the module.

  3. What does a passing line look like?

    Answer

    ok 1 - mean of 1 2 3

    The Test Anything Protocol form.

  4. What makes the build fail when a check fails?

    Answer

    ERROR STOP 1 in the report, so the status is 1

    The runner reads the status, not the text.

  5. What does assert_close take as its third argument?

    Answer

    The tolerance, so every test states its precision

    A hidden epsilon is a hidden assumption.

  6. Why does the module keep its counters PRIVATE?

    Answer

    So a test cannot reset or read them behind the assertions

    Only the four procedures are public.

  7. What does the detail after the colon give?

    Answer

    The expected and the actual value

    A failure a reader can act on.

  8. Which is not installed in this image?

    Answer

    pFUnit, test-drive and fortran-stdlib

    The page names them and shows no output of them.

  9. Why build the tests with -fcheck=all and -ffpe-trap?

    Answer

    A test that reads out of bounds and passes is worse than one that fails

    The test build is where the checks belong.

  10. What does GFORTRAN_ERROR_BACKTRACE=0 remove from the failing run?

    Answer

    The address list after ERROR STOP

    The message and the status stay.

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev