Comparing floating-point numbers

Two reals that a program computes are almost never equal, and the ones a program writes as literals are only equal if they came from the same text. This lesson takes the oldest example in floating point, runs it in single and in double precision with quite different answers, measures one unit in the last place with epsilon, and writes the test a program should use instead of ==.

The classic example

0.1 + 0.2 against 0.3 is the example every book uses because it shows the problem in two lines. The result depends on the precision, and this machine gives both answers: in single precision the two values are the same float, in double precision they are not, and the difference is the smallest amount a double can hold at that magnitude.

The file eq.f90:

program eqdemo use iso_fortran_env, only: real32, real64 implicit none real(real32) :: a, b, c, small real(real64) :: x, y, z a = 0.1_real32 b = 0.2_real32 c = a + b print '(a,es15.7)', 'single a + b = ', c print '(a,es15.7)', 'single 0.3 = ', 0.3_real32 print '(a,l1)', 'single equal = ', c == 0.3_real32 x = 0.1_real64 y = 0.2_real64 z = x + y print '(a,es26.17)', 'double x + y = ', z print '(a,es26.17)', 'double 0.3 = ', 0.3_real64 print '(a,es26.17)', 'double difference = ', abs(z - 0.3_real64) print '(a,l1)', 'double equal = ', z == 0.3_real64 small = epsilon(1.0_real32) print '(a,es15.7)', 'epsilon(1.0) = ', small print '(a,l1)', '1 + eps == 1 = ', 1.0_real32 + small == 1.0_real32 print '(a,l1)', '1 + eps/2 == 1 = ', 1.0_real32 + small / 2.0 == 1.0_real32 print '(a,l1)', 'nearly equal = ', nearly_equal(z, 0.3_real64, 1.0e-12_real64) print '(a,l1)', 'nearly equal bad = ', nearly_equal(z, 0.4_real64, 1.0e-12_real64) contains function nearly_equal(p, q, rtol) result(ok) real(real64), intent(in) :: p, q, rtol logical :: ok real(real64) :: scale scale = max(abs(p), abs(q)) ok = abs(p - q) <= rtol * max(scale, tiny(scale)) end function nearly_equal end program eqdemo

Each of the four equality comparisons is a warning under -Wall -Wextra, and the first command counts them:

gfortran -Wall -Wextra -std=f2018 eq.f90 -o eq 2>&1 | grep -c Warning

4

Four comparisons, four warnings: [-Wcompare-reals] is gfortran saying that an equality test on a real is usually a mistake. The build succeeds, and the program runs:

./eq

single a + b = 3.0000001E-01
single 0.3 = 3.0000001E-01
single equal = T
double x + y = 3.00000000000000044E-01
double 0.3 = 2.99999999999999989E-01
double difference = 5.55111512312578270E-17
double equal = F
epsilon(1.0) = 1.1920929E-07
1 + eps == 1 = F
1 + eps/2 == 1 = T
nearly equal = T
nearly equal bad = F

The single-precision pair is equal, and that is not a lucky accident: 0.1 and 0.2 each round to a single-precision float, their sum rounds back to the same float as 0.3 written with the real32 kind, and the rounding hides the error. In double precision the two sums differ in the last of the seventeen digits the es26.17 format prints, and equality fails. A program that tested the single-precision version, saw T and concluded that the comparison was safe would be wrong on the next machine with another precision.

EPSILON and one unit

epsilon(1.0) is the difference between one and the next float above it: one unit in the last place at the value one, 1.1920929E-07 in single precision. Adding it to one changes the value, and adding half of it does not, because the result has to round back to one of the two neighbours. That is the resolution of the type at that magnitude, and it is the reason a tolerance has to be relative: the same absolute step is huge near zero and invisible near a million.

The two lines are also the proof that a loop which adds a tiny increment to a large accumulator stops making progress. In single precision, adding 1.0E-08 to 1.0 leaves 1.0 for ever, and the loop runs to its limit instead of its sum.

An absolute and a relative tolerance

An absolute tolerance is a number in the units of the value: abs(p - q) <= 1.0e-6. It is right for a value that is near zero, a residual or a difference of two measured quantities, and wrong for a value whose magnitude nobody knows: a tolerance of 1e-6 accepts an error of a millionth of a metre and rejects a correct answer in kilometres.

A relative tolerance divides by the size of the values: abs(p - q) <= rtol * max(abs(p), abs(q)). That is the test the function uses, and it scales with the numbers. The max(scale, tiny(scale)) is the guard against a division by zero when both values are zero, which is the one case where a relative test has no meaning and an absolute one is needed.

The usual arrangement in a test suite is both: a value that may be near zero is tested with an absolute tolerance, everything else with a relative one, and a mixed test accepts a result that passes either. The tolerance itself is a property of the algorithm, not of the language: a solver that iterates to 1e-12 has no business being tested to 1e-15.

A nearly-equal function

The function in the program is the whole test in four lines: the scale is the larger magnitude of the two values, the difference is compared against the relative tolerance times that scale, and the guard keeps a pair of zeros from dividing by zero. It answers T for the sum of 0.1 and 0.2 against 0.3 with a tolerance of 1e-12, and F for the same sum against 0.4, which is the second property a test needs: it must also reject.

A test that only checks the passing case is not a test. Every tolerance-based comparison in a test suite should have a companion that is just outside the tolerance and must fail, or the suite passes for the wrong reason after a refactoring.

Turning the warning into an error

A project that has decided that == on reals is a bug can make the warning a build failure, and gfortran accepts the flag that names it:

program eqstrict use iso_fortran_env, only: real64 implicit none real(real64) :: x, y x = 0.1_real64 y = 0.2_real64 if (x + y == 0.3_real64) print *, 'equal' end program eqstrict

gfortran -Wall -Wextra -Werror=compare-reals -std=f2018 eqstrict.f90 -o eqstrict

eqstrict.f90:8:8:

8 | if (x + y == 0.3_real64) print *, 'equal'
| 1
Error: Equality comparison for REAL(8) at (1) [-Werror=compare-reals]
f951: some warnings being treated as errors

The second line, f951: some warnings being treated as errors, is how gfortran reports that the compile failed because of a warning. The flag is worth putting in a project that has no tolerance tests yet: it stops the next == before it becomes a test that passes on this machine and fails on the next.

The rule for a test

Four rules cover the comparisons this course needs. Compare with a tolerance, never with ==, except where exactness is the point, such as a counter or a bit pattern. Choose the tolerance from the algorithm and write it down, so that a reader knows what the test guarantees. Test one case inside the tolerance and one outside it. And when a comparison fails, print the difference and the tolerance rather than the two values: the two values are what the reader can already see.

The capstone tests of part 14 use exactly this function, and the testing group after the course adds a module that wraps it with a counter and an exit status. Nothing about it is special to Fortran except the warning that made the mistake visible in the first place.

Exercises

  1. Change the tolerance of the failing call from 1.0e-12 to 1.0e-2 and rerun. Which of the two answers changes, and what does that say about the test?
  2. Compare 0.1_real32 + 0.2_real32 with 0.3_real32 written through a variable rather than a literal. Is the answer still T?
  3. Write the absolute version of nearly_equal and use it on the difference of two values near zero. Which one passes where the relative version fails?
  4. Add -Werror=compare-reals to the build line of the main program. Which line of the program stops the build first?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. Why is == a poor test for two computed reals?

    Answer

    Rounding makes the two values differ in the last digits

    The difference is small but real, and equality is exact.

  2. What is epsilon(1.0)?

    Answer

    The difference between 1.0 and the next representable value

    It is one unit in the last place at the value one.

  3. What does this program print? program p use iso_fortran_env, only: real32 implicit none real(real32) :: e, one e = epsilon(1.0_real32) one = 1.0_real32 print *, one + e == one, one + e / 2.0 == one end program p

    Answer

    F T

    Half a unit rounds back to one; a whole unit does not.

  4. When is an absolute tolerance the right test?

    Answer

    For a value near zero, such as a residual

    A relative test has no meaning at zero.

  5. What does the guard max(scale, tiny(scale)) prevent?

    Answer

    A relative test with both values zero

    It keeps the scale positive when both values are zero.

  6. Why must a tolerance test also have a case that fails?

    Answer

    Otherwise it passes for the wrong reason after a change

    A test that cannot fail checks nothing.

  7. What does -Werror=compare-reals do?

    Answer

    Turns the equality-comparison warning into a build error

    The compile stops with that warning as an error.

  8. Why did the single-precision example report equality?

    Answer

    Both sums round to the same float, so the error is invisible

    The rounding hides a difference the double case shows.

  9. What should a failing comparison print?

    Answer

    The difference and the tolerance

    The difference against the tolerance is the diagnosis.

  10. What is a good way to pick the tolerance of a solver test?

    Answer

    From the convergence criterion of the algorithm

    A solver that iterates to 1e-12 cannot be tested to 1e-15.

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev