Comparing floating-point results

The second test a numerical program writes compares two reals, and it is the one that decides whether the suite is worth anything. Too strict and a correct answer fails on the last bit; too loose and a wrong answer passes. This page measures the three ways to state the distance between two values on six pairs from this machine, and gives the formula that a test should use.

The exact comparison is wrong

0.1_dp + 0.2_dp is not 0.3_dp in double precision, and sqrt(2.0_dp)**2 is not two. Both differences are one unit in the last place, which is the best any implementation can do, and a test written with == fails for a reason that has nothing to do with the code it is testing. gfortran even warns about it: -Wcompare-reals fires on every == and /= between reals, and the course pages that need one say so.

Three measures

Absolute error is abs(x - y). It is the distance in the units of the value, which is what a reader wants for a physical quantity and what nobody wants for a number that may be a million.

Relative error is abs(x - y) / max(abs(x), abs(y)). It is a fraction of the value, so 1.0e-12 means twelve correct digits at any magnitude, and it is the measure to state in an assertion. The division is what makes the zero case special.

ULP distance is abs(x - y) / spacing(max(abs(x), abs(y))): how many representable numbers lie between the two values. SPACING(x) is the gap between x and the next representable value, so one ULP is the smallest difference the type can express at that magnitude. It is the measure a library uses when it promises "correctly rounded" and the one that shows how a value at 1.0e10 can be one unit away and five hundred thousand ULPs away at the same time.

The combined test

A test wants one predicate, and the one that works for both a large value and a value near zero is the combination of a relative and an absolute term:

pure function close_enough(x, y, rtol, atol) result(ok) real(dp), intent(in) :: x, y, rtol, atol logical :: ok ok = abs(x - y) <= atol + rtol * max(abs(x), abs(y)) end function close_enough

The absolute term atol handles the values that are zero or near it, where a relative test divides by nothing useful. The relative term handles everything else, and the larger of the two magnitudes is used so that the test is symmetric: close_enough(a,b) and close_enough(b,a) agree, which a test written as abs(x-y)/abs(y) does not.

Six pairs from this machine

The program prints the two epsilons of the compiler and then six pairs of values, with the three measures and the verdict of three tests: an absolute one at 1.0e-15, a relative one at 1.0e-12, and the combined one with both.

program compare use, intrinsic :: iso_fortran_env, only: dp => real64, sp => real32 implicit none integer, parameter :: n = 6 real(dp) :: a(n), b(n) character(len=24) :: label(n) integer :: i label(1) = '0.1 + 0.2 vs 0.3' label(2) = 'sqrt(2)**2 vs 2' label(3) = '1.0/3.0 vs 0.333...' label(4) = '1e10 + 1 vs 1e10' label(5) = '0.0 vs 1e-20' label(6) = 'sin(pi) vs 0' a(1) = 0.1_dp + 0.2_dp b(1) = 0.3_dp a(2) = sqrt(2.0_dp)**2 b(2) = 2.0_dp a(3) = 1.0_dp / 3.0_dp b(3) = 0.3333333333333333_dp a(4) = 1.0e10_dp + 1.0_dp b(4) = 1.0e10_dp a(5) = 0.0_dp b(5) = 1.0e-20_dp a(6) = sin(acos(-1.0_dp)) b(6) = 0.0_dp print '(a,es12.3,a)', 'epsilon(double) = ', epsilon(1.0_dp), ' (the gap at 1.0)' print '(a,es12.3,a)', 'epsilon(single) = ', epsilon(1.0_sp), ' (the gap at 1.0)' print * print '(a)', 'pair value abs err rel err ulps abs-only rel-only both' do i = 1, n print '(a24,2x,es20.13,2x,es9.2,2x,es9.2,2x,es9.2,2x,l7,2x,l7,2x,l5)', label(i), a(i), abs(a(i) - b(i)), & rel_err(a(i), b(i)), ulps(a(i), b(i)), abs(a(i) - b(i)) <= 1.0e-15_dp, & rel_err(a(i), b(i)) <= 1.0e-12_dp, close_enough(a(i), b(i), 1.0e-12_dp, 1.0e-15_dp) end do contains pure function rel_err(x, y) result(e) real(dp), intent(in) :: x, y real(dp) :: e real(dp) :: scale scale = max(abs(x), abs(y)) if (scale > 0.0_dp) then e = abs(x - y) / scale else e = 0.0_dp end if end function rel_err pure function ulps(x, y) result(near) real(dp), intent(in) :: x, y real(dp) :: near real(dp) :: gap gap = spacing(max(abs(x), abs(y))) near = abs(x - y) / gap end function ulps pure function close_enough(x, y, rtol, atol) result(ok) real(dp), intent(in) :: x, y, rtol, atol logical :: ok ok = abs(x - y) <= atol + rtol * max(abs(x), abs(y)) end function close_enough end program compare

gfortran -Wall -Wextra -std=f2018 compare.f90 -o compare
./compare

epsilon(double) =    2.220E-16   (the gap at 1.0)
epsilon(single) =    1.192E-07   (the gap at 1.0)

pair                     value                 abs err    rel err    ulps  abs-only  rel-only  both
0.1 + 0.2 vs 0.3           3.0000000000000E-01   5.55E-17   1.85E-16   1.00E+00        T        T      T
sqrt(2)**2 vs 2            2.0000000000000E+00   4.44E-16   2.22E-16   1.00E+00        T        T      T
1.0/3.0 vs 0.333...        3.3333333333333E-01   0.00E+00   0.00E+00   0.00E+00        T        T      T
1e10 + 1 vs 1e10           1.0000000001000E+10   1.00E+00   1.00E-10   5.24E+05        F        F      F
0.0 vs 1e-20               0.0000000000000E+00   1.00E-20   1.00E+00   6.65E+15        T        F      T
sin(pi) vs 0               1.2246467991474E-16   1.22E-16   1.00E+00   4.97E+15        T        F      T

Four rows of that table say something a test writer needs. The first two are the ordinary case: one ULP, and every reasonable test passes. The third is the value of comparing against a decimal literal: 1.0_dp/3.0_dp and 0.3333333333333333_dp are the same double, so the distance is exactly zero. The fourth is a value at 1.0e10: the absolute error is one, the relative error is 1.0e-10, and the ULP distance is 5.24e+05, which is why a test that states ULPs must state them per magnitude and a test that states a relative error does not have to.

When the reference is zero

The last two rows are the case that breaks a relative-only test. sin(pi) is 1.22e-16 rather than zero, and its relative error against zero is one: a hundred per cent. The combined test passes it because the absolute error is below 1.0e-15, which is the right answer: a value of 1e-16 where the mathematics says zero is as close to zero as double precision gets.

atol is therefore not a decoration. It has to be chosen from the scale of the problem: a computation of a length in metres does not want the same absolute tolerance as a computation of a probability, and a program whose results are all near zero needs the absolute term to carry the test.

What to put in an assertion

For a single value, use the combined test with a relative tolerance picked from the precision of the type and the conditioning of the computation, and write the absolute one beside it. For a vector or a matrix, compare the maximum of the componentwise relative errors, as the matrix capstone does, so that one bad component fails the test. For an iteration, compare the residual rather than the step, as the solver capstone does, so that a method which wanders does not report success.

The assertion module of the previous page takes the relative tolerance as an argument and compares with exactly the formula above, which is why the two pages belong together: the formula is here, the assertion that uses it is there.

Exercises

  1. Add a row for exp(log(10.0_dp)) against ten. Which measure is smallest, and why?
  2. Compute the ULP distance between 1.0_dp and 1.0_dp + epsilon(1.0_dp), then between 1.0e10_dp and 1.0e10_dp + epsilon(1.0_dp). What does the second number mean?
  3. Write the same six pairs in single precision and compare the table. Which rows change their verdict?
  4. Set atol to zero and rerun. Which rows fail, and would a reader call those failures bugs?

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev