A statistics module

The statistics of a list are four lines each and a classic place to learn what a module is for: every program that reads a table of numbers wants the same mean, the same variance and the same least-squares line, and every one of them should get the same answer. This lesson writes the module, prints its values for a small dataset, and puts a driver behind it that checks each number against a tolerance and counts the failures.

The module

Three pure functions and one subroutine, all of them generic over the size of the array and written against assumed-shape dummies, so a caller passes any length. The functions are pure because they only read their arguments, which lets the compiler keep them in a parallel loop later.

The file stats.f90:

module stats_mod implicit none private public :: mean, variance, stddev, linfit contains pure function mean(x) result(m) real, intent(in) :: x(:) real :: m m = sum(x) / real(size(x)) end function mean pure function variance(x) result(v) real, intent(in) :: x(:) real :: v, m m = mean(x) v = sum((x - m) ** 2) / real(size(x) - 1) end function variance pure function stddev(x) result(s) real, intent(in) :: x(:) real :: s s = sqrt(variance(x)) end function stddev subroutine linfit(x, y, slope, intercept, rms) real, intent(in) :: x(:), y(:) real, intent(out) :: slope, intercept, rms real :: mx, my, sxy, sxx integer :: i mx = mean(x) my = mean(y) sxy = 0.0 sxx = 0.0 do i = 1, size(x) sxy = sxy + (x(i) - mx) * (y(i) - my) sxx = sxx + (x(i) - mx) ** 2 end do slope = sxy / sxx intercept = my - slope * mx rms = 0.0 do i = 1, size(x) rms = rms + (y(i) - (slope * x(i) + intercept)) ** 2 end do rms = sqrt(rms / real(size(x))) end subroutine linfit end module stats_mod program statsdemo use stats_mod implicit none real :: data(6), x(5), y(5), slope, intercept, rms integer :: nfail data = [2.0, 4.0, 4.0, 4.0, 5.0, 9.0] print '(a,f10.4)', 'mean = ', mean(data) print '(a,f10.4)', 'variance = ', variance(data) print '(a,f10.4)', 'stddev = ', stddev(data) x = [1.0, 2.0, 3.0, 4.0, 5.0] y = [2.1, 3.9, 6.2, 7.8, 10.1] call linfit(x, y, slope, intercept, rms) print '(a,f8.4)', 'slope = ', slope print '(a,f8.4)', 'intercept= ', intercept print '(a,f8.4)', 'rms = ', rms nfail = 0 call check('mean', mean(data), 4.6667, 1.0e-3) call check('variance', variance(data), 5.4667, 1.0e-3) call check('stddev', stddev(data), 2.3381, 1.0e-3) call check('slope', slope, 1.9900, 1.0e-3) call check('intercept', intercept, 0.0500, 1.0e-3) print '(a,i0,a,i0)', 'checks: ', 5, ' failed: ', nfail contains subroutine check(name, got, want, tol) character(len=*), intent(in) :: name real, intent(in) :: got, want, tol if (abs(got - want) <= tol) then print '(a,a,a,f9.4,a,f9.4)', 'PASS ', name, ' got ', got, ' want ', want else nfail = nfail + 1 print '(a,a,a,f9.4,a,f9.4)', 'FAIL ', name, ' got ', got, ' want ', want end if end subroutine check end program statsdemo

gfortran -Wall -Wextra -std=f2018 stats.f90 -o stats
./stats

mean = 4.6667
variance = 5.4667
stddev = 2.3381
slope = 1.9900
intercept= 0.0500
rms = 0.1463
PASS mean got 4.6667 want 4.6667
PASS variance got 5.4667 want 5.4667
PASS stddev got 2.3381 want 2.3381
PASS slope got 1.9900 want 1.9900
PASS intercept got 0.0500 want 0.0500
checks: 5 failed: 0

Mean, variance and deviation

The mean is sum(x) / size(x), one intrinsic and one division. The variance is the mean of the squared deviations, and the module writes it as sum((x - m)**2) / (n - 1): one whole-array expression for the deviations, reduced by sum, with no loop at all. The standard deviation is its square root, and the three functions compose without a temporary array longer than the expression itself.

For the dataset 2, 4, 4, 4, 5, 9 the mean is 28/6 = 4.6667 and the variance 5.4667, which is 32.8/6 with the divisor of the sample form. A calculator that uses the population form gives 4.5556, and a test that compares the two without knowing which one the code uses fails for a reason that has nothing to do with the code.

Why n minus one

With n in the divisor the variance of a sample underestimates the variance of the population it came from, because the sample mean was fitted to the same data and the deviations are measured from a point that already moved towards them. The n - 1 corrects for that fitted parameter and makes the estimator unbiased. It is called Bessel's correction, and the module names its function variance without saying which one it computes, which a real library would have to.

For one value the sample variance divides by zero, and the function returns an infinity or a NaN depending on the arithmetic: a library would check size(x) > 1 and report an error state. A course module can leave it, a program that processes measurements cannot.

A least-squares line

The fit is the closed form every first course derives: the slope is the covariance of x and y over the variance of x, the intercept makes the line pass through the point of the two means, and the rms is the root of the mean squared residual. For the five points of the program the slope is 1.99 and the intercept 0.05, which is a line through 2.04 at x = 1 and 10.00 at x = 5: the data were built to be that line plus a small wobble, and the rms of 0.1463 is the size of the wobble.

The formula is stable only for x values that are not all nearly equal, because the denominator is the spread of x. A fit through points measured at the same x is not a fit at all, and the module would divide by a number near zero without complaint. That is the same class of check as the singular pivot of lesson 095, and it belongs at the same level: a library reports it, a lesson names it.

The checks

Five calls, five values, one tolerance, and a counter. The subroutine is a contained procedure so that it sees nfail and can add to it; a real test driver passes the counter as an intent(inout) argument, which is what the testing group after this course does. The tolerance is 1e-3, which is loose enough for the four decimals printed and tight enough to catch a wrong formula.

The expected values are written as literals in the test, not computed from the same functions, which is the point: a test that recomputes the answer with the code under test passes whatever the code does. Each expected value would come from a worked example, a textbook or a second implementation, and it is written down here as the promise the module has to keep.

An integer array

The dummies are single precision, so an integer dataset is refused rather than converted silently:

module stats_mod implicit none contains pure function mean(x) result(m) real, intent(in) :: x(:) real :: m m = sum(x) / real(size(x)) end function mean end module stats_mod program badstat use stats_mod implicit none integer :: data(4) data = [2, 4, 4, 9] print '(a,f10.4)', 'mean = ', mean(data) end program badstat

gfortran -Wall -Wextra -std=f2018 badstat.f90 -o badstat

badstat.f90:17:31:

17 | print '(a,f10.4)', 'mean = ', mean(data)
| 1
Error: Type mismatch in argument 'x' at (1); passed INTEGER(4) to REAL(4)

The compiler says which dummy and both types, and the fix is either a conversion at the call site, real(data), which makes a temporary array, or a second specific behind a generic mean that does the conversion inside. For a dataset of integers the first is fine; for a library the generic is the reason lesson 066 exists.

Why a module

Four procedures that every caller would otherwise write for itself, one definition of what variance means, and an explicit interface that turns a wrong argument into a compile error. The module also fixes the precision: a caller that needs double precision needs another module or a generic over kinds, and a program that mixes the two is a program with a subtle loss of digits rather than an error.

The next lesson takes these numbers and prints them as a report with a table, which is the other half of the work: the arithmetic here and the formatting there. The old capstone this lesson replaces did both at once, and the course now separates them because a report is a reader's problem and a statistic is a mathematician's.

Exercises

  1. Add a median function. What has to happen to the input before the middle value can be taken, and does it change the caller's array?
  2. Change the divisor of variance to n and rerun the checks. Which two lines fail?
  3. Make linfit return the correlation coefficient as well. Which formula do you need, and which two sums are already there?
  4. Call variance with a one-element array and print the result. What does the program print, and why is it not an error?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does the sample variance divide by?

    Answer

    n - 1, Bessel&#39;s correction for the fitted mean

    The correction makes the estimator unbiased.

  2. Why are the statistics functions PURE?

    Answer

    They only read their arguments, so the compiler may schedule them freely

    A pure function may not change anything outside itself.

  3. What does this program print? program p implicit none real :: x(4) x = [1.0, 2.0, 3.0, 4.0] print *, sum(x) / size(x) end program p

    Answer

    2.5

    Ten over four is two and a half.

  4. What is the slope of a least-squares line?

    Answer

    The covariance of x and y over the variance of x

    The closed form the module implements.

  5. Why are the expected values of the checks written as literals?

    Answer

    A test that recomputes them with the same code proves nothing

    The expected value is an independent promise.

  6. What happens to the variance of a one-element array?

    Answer

    It divides by zero and returns an infinity or a NaN

    A library would test the size and report an error state.

  7. When is the least-squares formula unstable?

    Answer

    When all the x values are nearly equal

    The denominator is the spread of x.

  8. What does gfortran report for an integer array passed to mean?

    Answer

    Error: type mismatch in argument; passed INTEGER(4) to REAL(4)

    The dummy is single precision and no conversion is implied.

  9. What is the rms of the residual in the fit?

    Answer

    The square root of the mean of the squared residuals

    It is the size of the wobble around the line.

  10. Why does the module fix the precision of its functions?

    Answer

    A caller with another kind needs a generic or another module

    Mixing kinds loses digits silently instead of failing.

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev