A statistics module
The statistics of a list are four lines each and a classic place to learn what a module is for: every program that reads a table of numbers wants the same mean, the same variance and the same least-squares line, and every one of them should get the same answer. This lesson writes the module, prints its values for a small dataset, and puts a driver behind it that checks each number against a tolerance and counts the failures.
The module
Three pure functions and one subroutine, all of them generic over the size of the array and written against assumed-shape dummies, so a caller passes any length. The functions are pure because they only read their arguments, which lets the compiler keep them in a parallel loop later.
The file stats.f90:
module stats_mod
implicit none
private
public :: mean, variance, stddev, linfit
contains
pure function mean(x) result(m)
real, intent(in) :: x(:)
real :: m
m = sum(x) / real(size(x))
end function mean
pure function variance(x) result(v)
real, intent(in) :: x(:)
real :: v, m
m = mean(x)
v = sum((x - m) ** 2) / real(size(x) - 1)
end function variance
pure function stddev(x) result(s)
real, intent(in) :: x(:)
real :: s
s = sqrt(variance(x))
end function stddev
subroutine linfit(x, y, slope, intercept, rms)
real, intent(in) :: x(:), y(:)
real, intent(out) :: slope, intercept, rms
real :: mx, my, sxy, sxx
integer :: i
mx = mean(x)
my = mean(y)
sxy = 0.0
sxx = 0.0
do i = 1, size(x)
sxy = sxy + (x(i) - mx) * (y(i) - my)
sxx = sxx + (x(i) - mx) ** 2
end do
slope = sxy / sxx
intercept = my - slope * mx
rms = 0.0
do i = 1, size(x)
rms = rms + (y(i) - (slope * x(i) + intercept)) ** 2
end do
rms = sqrt(rms / real(size(x)))
end subroutine linfit
end module stats_mod
program statsdemo
use stats_mod
implicit none
real :: data(6), x(5), y(5), slope, intercept, rms
integer :: nfail
data = [2.0, 4.0, 4.0, 4.0, 5.0, 9.0]
print '(a,f10.4)', 'mean = ', mean(data)
print '(a,f10.4)', 'variance = ', variance(data)
print '(a,f10.4)', 'stddev = ', stddev(data)
x = [1.0, 2.0, 3.0, 4.0, 5.0]
y = [2.1, 3.9, 6.2, 7.8, 10.1]
call linfit(x, y, slope, intercept, rms)
print '(a,f8.4)', 'slope = ', slope
print '(a,f8.4)', 'intercept= ', intercept
print '(a,f8.4)', 'rms = ', rms
nfail = 0
call check('mean', mean(data), 4.6667, 1.0e-3)
call check('variance', variance(data), 5.4667, 1.0e-3)
call check('stddev', stddev(data), 2.3381, 1.0e-3)
call check('slope', slope, 1.9900, 1.0e-3)
call check('intercept', intercept, 0.0500, 1.0e-3)
print '(a,i0,a,i0)', 'checks: ', 5, ' failed: ', nfail
contains
subroutine check(name, got, want, tol)
character(len=*), intent(in) :: name
real, intent(in) :: got, want, tol
if (abs(got - want) <= tol) then
print '(a,a,a,f9.4,a,f9.4)', 'PASS ', name, ' got ', got, ' want ', want
else
nfail = nfail + 1
print '(a,a,a,f9.4,a,f9.4)', 'FAIL ', name, ' got ', got, ' want ', want
end if
end subroutine check
end program statsdemo
gfortran -Wall -Wextra -std=f2018 stats.f90 -o stats
./stats
mean = 4.6667
variance = 5.4667
stddev = 2.3381
slope = 1.9900
intercept= 0.0500
rms = 0.1463
PASS mean got 4.6667 want 4.6667
PASS variance got 5.4667 want 5.4667
PASS stddev got 2.3381 want 2.3381
PASS slope got 1.9900 want 1.9900
PASS intercept got 0.0500 want 0.0500
checks: 5 failed: 0
Mean, variance and deviation
The mean is sum(x) / size(x), one intrinsic and one division. The variance is the mean of the squared deviations, and the module writes it as sum((x - m)**2) / (n - 1): one whole-array expression for the deviations, reduced by sum, with no loop at all. The standard deviation is its square root, and the three functions compose without a temporary array longer than the expression itself.
For the dataset 2, 4, 4, 4, 5, 9 the mean is 28/6 = 4.6667 and the variance 5.4667, which is 32.8/6 with the divisor of the sample form. A calculator that uses the population form gives 4.5556, and a test that compares the two without knowing which one the code uses fails for a reason that has nothing to do with the code.
Why n minus one
With n in the divisor the variance of a sample underestimates the variance of the population it came from, because the sample mean was fitted to the same data and the deviations are measured from a point that already moved towards them. The n - 1 corrects for that fitted parameter and makes the estimator unbiased. It is called Bessel's correction, and the module names its function variance without saying which one it computes, which a real library would have to.
For one value the sample variance divides by zero, and the function returns an infinity or a NaN depending on the arithmetic: a library would check size(x) > 1 and report an error state. A course module can leave it, a program that processes measurements cannot.
A least-squares line
The fit is the closed form every first course derives: the slope is the covariance of x and y over the variance of x, the intercept makes the line pass through the point of the two means, and the rms is the root of the mean squared residual. For the five points of the program the slope is 1.99 and the intercept 0.05, which is a line through 2.04 at x = 1 and 10.00 at x = 5: the data were built to be that line plus a small wobble, and the rms of 0.1463 is the size of the wobble.
The formula is stable only for x values that are not all nearly equal, because the denominator is the spread of x. A fit through points measured at the same x is not a fit at all, and the module would divide by a number near zero without complaint. That is the same class of check as the singular pivot of lesson 095, and it belongs at the same level: a library reports it, a lesson names it.
The checks
Five calls, five values, one tolerance, and a counter. The subroutine is a contained procedure so that it sees nfail and can add to it; a real test driver passes the counter as an intent(inout) argument, which is what the testing group after this course does. The tolerance is 1e-3, which is loose enough for the four decimals printed and tight enough to catch a wrong formula.
The expected values are written as literals in the test, not computed from the same functions, which is the point: a test that recomputes the answer with the code under test passes whatever the code does. Each expected value would come from a worked example, a textbook or a second implementation, and it is written down here as the promise the module has to keep.
An integer array
The dummies are single precision, so an integer dataset is refused rather than converted silently:
module stats_mod
implicit none
contains
pure function mean(x) result(m)
real, intent(in) :: x(:)
real :: m
m = sum(x) / real(size(x))
end function mean
end module stats_mod
program badstat
use stats_mod
implicit none
integer :: data(4)
data = [2, 4, 4, 9]
print '(a,f10.4)', 'mean = ', mean(data)
end program badstat
gfortran -Wall -Wextra -std=f2018 badstat.f90 -o badstat
badstat.f90:17:31:
17 | print '(a,f10.4)', 'mean = ', mean(data)
| 1
Error: Type mismatch in argument 'x' at (1); passed INTEGER(4) to REAL(4)
The compiler says which dummy and both types, and the fix is either a conversion at the call site, real(data), which makes a temporary array, or a second specific behind a generic mean that does the conversion inside. For a dataset of integers the first is fine; for a library the generic is the reason lesson 066 exists.
Why a module
Four procedures that every caller would otherwise write for itself, one definition of what variance means, and an explicit interface that turns a wrong argument into a compile error. The module also fixes the precision: a caller that needs double precision needs another module or a generic over kinds, and a program that mixes the two is a program with a subtle loss of digits rather than an error.
The next lesson takes these numbers and prints them as a report with a table, which is the other half of the work: the arithmetic here and the formatting there. The old capstone this lesson replaces did both at once, and the course now separates them because a report is a reader's problem and a statistic is a mathematician's.
Exercises
- Add a median function. What has to happen to the input before the middle value can be taken, and does it change the caller's array?
- Change the divisor of variance to n and rerun the checks. Which two lines fail?
- Make linfit return the correlation coefficient as well. Which formula do you need, and which two sums are already there?
- Call variance with a one-element array and print the result. What does the program print, and why is it not an error?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does the sample variance divide by?
Answer
n - 1, Bessel's correction for the fitted mean
The correction makes the estimator unbiased.
-
Why are the statistics functions PURE?
Answer
They only read their arguments, so the compiler may schedule them freely
A pure function may not change anything outside itself.
-
What does this program print?
program p implicit none real :: x(4) x = [1.0, 2.0, 3.0, 4.0] print *, sum(x) / size(x) end program pAnswer
2.5
Ten over four is two and a half.
-
What is the slope of a least-squares line?
Answer
The covariance of x and y over the variance of x
The closed form the module implements.
-
Why are the expected values of the checks written as literals?
Answer
A test that recomputes them with the same code proves nothing
The expected value is an independent promise.
-
What happens to the variance of a one-element array?
Answer
It divides by zero and returns an infinity or a NaN
A library would test the size and report an error state.
-
When is the least-squares formula unstable?
Answer
When all the x values are nearly equal
The denominator is the spread of x.
-
What does gfortran report for an integer array passed to
mean?Answer
Error: type mismatch in argument; passed INTEGER(4) to REAL(4)
The dummy is single precision and no conversion is implied.
-
What is the rms of the residual in the fit?
Answer
The square root of the mean of the squared residuals
It is the size of the wobble around the line.
-
Why does the module fix the precision of its functions?
Answer
A caller with another kind needs a generic or another module
Mixing kinds loses digits silently instead of failing.
The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev