Tests on both sides
A project with two languages needs two suites and one status. The Fortran suite checks the arithmetic through the same interface the world sees; the Python suites check the wrapper and the service. Neither replaces the other, and this page says exactly what each one proves.
Two suites, one project
Nineteen checks in three files: six Fortran checks in one program, nine unittest cases for the wrapper and four for the service. The Fortran program cannot test the wrapper, because it does not exist in that process; the Python tests cannot test the arithmetic independently, because they ask the same library. What they can do is test the two layers separately, and that is the division of labour below.
The Fortran suite
The program was shown on the core page: it declares the four functions with BIND(C) in an INTERFACE block and calls the library the way a C program would. It is a client, not a USE of the module, and that is what makes it a test of the exported interface rather than of the source.
./test_core.sh
built libstats.so
ok 1 - mean of four values
ok 2 - sd of four values
ok 3 - median of four values
ok 4 - quantile 0.5 of four values
ok 5 - quantile status is 0
ok 6 - quantile status is 1 for q > 1
# 6 checks, 0 failures
Line 5 and line 6 are the two halves of the status contract: a call with a good quantile returns 0, and a call with q = 1.5 returns 1 and leaves the output at zero. That is the one behaviour the Python side turns into an exception, so it is worth testing on this side of the boundary where the value is visible.
The Python suites
Nine cases for the wrapper and four for the service, both on the standard library's unittest. The wrapper cases are the arithmetic and the two refusals; the service cases are the four responses of the JSON contract, against a real server on a port the system chooses. Both files were shown on the two previous pages.
cd python
python3 -m unittest
.............
----------------------------------------------------------------------
Ran 13 tests in 0.544s
OK
Thirteen dots, one line of summary and a status of zero. A failing test prints the traceback of the assertion and changes the status to one, which is the only part a build reads.
One runner
The runner is nine lines and it is the command a CI job calls. run-tests.sh:
#!/bin/sh # Run both sides of the test suite and keep the worst status. set -e cd "$(dirname "$0")" ./test_core.sh cd python python3 -m unittest
./run-tests.sh
echo $?
built libstats.so
ok 1 - mean of four values
ok 2 - sd of four values
ok 3 - median of four values
ok 4 - quantile 0.5 of four values
ok 5 - quantile status is 0
ok 6 - quantile status is 1 for q > 1
# 6 checks, 0 failures
.............
----------------------------------------------------------------------
Ran 13 tests in 0.544s
OK
0
set -e is what keeps the status: the script stops at the first command that fails, so a failing Fortran test never reaches Python and the shell returns the status of the failure. A project that wanted to run both suites and report both failures would drop set -e and remember the worst status, as the course's testing group does for several programs.
What each side proves
| check | what it proves | what it cannot see |
|---|---|---|
| the Fortran test program | the arithmetic, and that the four symbols exist with the C calling convention | the Python wrapper and its argument conversions |
| the wrapper tests | argtypes, restype, the array copy and the two refusals | a wrong number inside the core |
| the service tests | routing, JSON encoding, status codes, and a real socket | anything about the core the wrapper already got wrong |
The last column is the reason all three exist. A bug in the Fortran core fails the Fortran test and the wrapper test together, which is the correct signal: the number changed. A bug in the wrapper fails only the wrapper test. A bug in the routing fails only the service test. When two suites fail at once, the cause is on the lower layer.
What none of them does is test the numbers against an independent implementation. That is the separate check on the wrapper page: its numpy section compares the wrapper's mean and standard deviation with numpy's own, and that run passed with rtol=1e-12. The expected values in both suites are still the ones the algorithm was written to produce, which catches a change and not a mistake that was in the algorithm from the start; the numpy run covers the mean and the spread, and is what closes that gap for them.
Exercises
- Break the mean on purpose in the core and run the runner. Which suites fail, and what is the first failing line?
- Make run-tests.sh run both suites even when the first fails. What does it have to remember, and what status does it return?
- Add a test that calls the Fortran test program from Python and checks its status. What does that prove that the wrapper tests do not?
- Time the nineteen checks. Which one dominates, and would you keep it in the fast suite?
The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run with Python 3.14.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev