A text pipeline

The input

A pipeline step is a program that reads a file, does one job and prints a result another program can read. The input here is a small access log, one request per line, with an occasional line that is not a request:

2026-11-05 12:00:01 GET /api/users 200 12 2026-11-05 12:00:02 GET /api/users 500 31 2026-11-05 12:00:03 POST /api/orders 502 44 2026-11-05 12:00:04 GET /api/users 200 9 2026-11-05 12:00:05 GET /health 200 1 2026-11-05 12:00:06 broken line 2026-11-05 12:00:07 GET /api/orders 500 27

The job is the one from a real incident: which paths fail, and how often. A line that does not parse is counted separately instead of being silently dropped, because a parser that hides bad input hides the bug.

The script

File.foreach reads one line at a time, so the size of the log does not matter. The pattern is the validating regex from lesson 124; the hash counts the paths with a 5xx status:

pattern = /\A(?<time>\S+ \S+) (?<method>[A-Z]+) (?<path>\S+) (?<status>\d{3}) (?<ms>\d+)\z/ errors = Hash.new(0) total = 0 skipped = 0 File.foreach(ARGV.fetch(0)) do |line| total += 1 m = line.chomp.match(pattern) if m.nil? skipped += 1 next end next unless m[:status].start_with?("5") errors[m[:path]] += 1 end puts "lines: #{total}, skipped: #{skipped}" errors.sort_by { |path, count| [-count, path] }.each do |path, count| puts format("%-12s %d", path, count) end

Two details carry a pipeline: ARGV.fetch(0) makes a missing argument an error instead of reading standard input by accident, and sort_by with a two-element key makes the output stable even when two paths have the same count.

Running it

ruby filter.rb access.log

lines: 7, skipped: 1 /api/orders 2 /api/users 1

The summary line goes first, so a reader sees that one line was not counted. The two failing paths follow, most frequent first, which is the order an on-call engineer reads.

In a pipeline the same script is a stage whose output is captured to a file, and a CI job diffs that file against a known good one. The exit code is the other interface; a version that must fail the build when errors are present ends with exit 1 unless errors.empty?.

The shape of a pipeline step

• One input. A file argument, not a guess. Standard input is a separate mode when the tool needs it.
• One job. Filter, count or convert; not all three in one script.
• Count what you skip. A line that does not parse is a number in the report, not a silent next.
• Stable output. Sorted keys and fixed widths, so a diff between two runs means a real change.

What to watch for

line.chomp is required before \z: the newline at the end of a line otherwise fails the anchor. The alternative is chomp: true on readlines, which foreach does not take.

A hash with a default of zero is the counter; a hash of lists is for a report that needs the lines themselves, and it holds every matched line in memory. Pick the aggregate the report prints, not the data behind it.

Next: Formatting tables in the terminal

Article author: Arthur Isaev