Capstone: log report

The report

The second capstone reads an access log and prints a short report: how many requests, which status codes, which paths are popular, and how much data went out. It is the kind of script people write once a week at work.

Put a dozen combined-format lines in a file first, so every command on this page repeats the same run:

printf '127.0.0.1 - - [27/Oct/2026:10:14:02 +0000] "GET /dev/perl/ HTTP/1.1" 200 48710 "-" "-"\n127.0.0.1 - - [27/Oct/2026:10:14:03 +0000] "GET /dev/perl/course/ HTTP/1.1" 200 61234 "-" "-"\n127.0.0.1 - - [27/Oct/2026:10:14:04 +0000] "GET /dev/perl/course/ HTTP/1.1" 200 61234 "-" "-"\n10.0.0.7 - - [27/Oct/2026:10:14:06 +0000] "GET /dev/perl/ HTTP/1.1" 200 48710 "-" "-"\n10.0.0.7 - - [27/Oct/2026:10:14:07 +0000] "GET /dev/ruby/course/ HTTP/1.1" 200 51208 "-" "-"\n10.0.0.7 - - [27/Oct/2026:10:14:08 +0000] "GET /dev/perl/ HTTP/1.1" 200 48710 "-" "-"\n' > access.log

printf '10.0.0.9 - - [27/Oct/2026:10:14:11 +0000] "GET /dev/perl/course/ HTTP/1.1" 200 61234 "-" "-"\n10.0.0.9 - - [27/Oct/2026:10:14:12 +0000] "GET /dev/ruby/course/ HTTP/1.1" 200 51208 "-" "-"\n10.0.0.9 - - [27/Oct/2026:10:14:15 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "-"\n10.0.0.9 - - [27/Oct/2026:10:14:16 +0000] "POST /dev/perl/course/ HTTP/1.1" 500 - "-" "-"\n10.0.0.9 - - [27/Oct/2026:10:14:18 +0000] "GET /dev/perl/course/ HTTP/1.1" 200 61234 "-" "-"\n10.0.0.9 - - [27/Oct/2026:10:14:21 +0000] "GET /dev/perl/cour\n' >> access.log

The fragments in this lesson make one program, logreport. Assembled and made executable, it prints this for the log above:

./logreport access.log

requests: 11 errors: 2 (18.2%)
bytes: 0.5 MB

status
500 1
404 1
200 9

top paths
5 /dev/perl/course/
3 /dev/perl/
2 /dev/ruby/course/
1 /favicon.ico

The script reads any number of files, or standard input when given a dash, so it composes with zcat, ssh and find. It never stores the log in memory, only the counters.

Parsing a log line

A combined-format line has a fixed shape: host, identity, user, a bracketed time, a quoted request, a status, a byte count, and two quoted strings. One regular expression with the x flag names each part.

127.0.0.1 - - [27/Oct/2026:10:14:02 +0000] "GET /dev/perl/ HTTP/1.1" 200 5123 "-" "Mozilla/5.0"

my $re = qr{ ^(\S+) # client address \s+\S+\s+\S+ # identity and user \s+\[([^\]]+)\] # time, without brackets \s+"(\S+)\s+(\S+)[^"]*" # method and path \s+(\d{3})\s+(\S+) # status and bytes }x;

\S+ stops at whitespace, which is why the quoted request needs its own pattern: the path never contains a space in a valid log, but the protocol and the quoted strings after it do.

while (my $line = <>) { next unless $line =~ $re; my ($host, $time, $method, $path, $status, $bytes) = ($1, $2, $3, $4, $5, $6); $requests++; $status{$status}++; $paths{$path}++; $total_bytes += $bytes if $bytes =~ /^\d+$/; }

A dash is a legal byte count for a request that returned no body, so the numeric check matters. Without it, += on "-" warns and adds zero.

Aggregating and printing

With the counters collected, the report is sorting and formatting. Sort paths by count, take the first ten, and print the result in two aligned columns:

my $errors = 0; $errors += $status{$_} for grep { $_ >= 400 } keys %status; printf "requests: %d errors: %d (%.1f%%)\n", $requests, $errors, $requests ? 100 * $errors / $requests : 0; printf "bytes: %.1f MB\n\n", $total_bytes / 1024 ** 2; print "status\n"; printf " %d %d\n", $_, $status{$_} for sort { $b <=> $a } keys %status; my @top = (sort { $paths{$b} <=> $paths{$a} } keys %paths)[0 .. 9]; print "\ntop paths\n"; printf " %6d %s\n", $paths{$_}, $_ for grep { defined } @top;

Slicing [0 .. 9] on a short list leaves undef entries, hence the grep { defined } from the grep lesson. Ties in the sort come out in hash order, which varies between runs; add the path as a secondary key if you need a stable report.

The status table follows its own comparator, $b <=> $a on the codes, so the highest code is printed first. Swap $a and $b there if you would rather read the table from 200 upwards.

Bad lines and gzip

Real logs contain truncated lines, request lines from bots, and occasional binary garbage. Count what you skip and print the count at the end: a sudden rise means the log format changed, not that the script is broken.

if ($line !~ $re) { $skipped++; next; }

Rotated logs are usually compressed. IO::Uncompress::Gunzip opens a .gz file as a normal handle, so the parsing loop does not change:

use IO::Uncompress::Gunzip qw(gunzip $GunzipError); my $fh = IO::Uncompress::Gunzip->new($file) or die "cannot open $file: $GunzipError\n"; while (my $line = <$fh>) { print $line if $line =~ /\s500\s/; }

The same pattern works for .bz2 and .xz through IO::Uncompress::Bunzip2 and IO::Uncompress::UnXz. Combine that with Find to walk a log directory and you have a report for the whole month.

Finish the script with a --top N option and a --since filter on the timestamp. Both reuse Getopt::Long and the time parsing you already have, and both make the tool useful to someone other than its author.

Next: Performance and profiling

Article author: Arthur Isaev