Project: a link checker

The checker

The second project checks a list of URLs and fails when one of them is broken, which makes it a pipeline step. Each URL gets a thread, so a slow host does not hold up the rest, and the results are collected by position so the report keeps the order of the file.

#!/usr/bin/env ruby require "net/http" urls = File.readlines(ARGV[0], chomp: true).reject(&:empty?) results = Array.new(urls.length) threads = urls.each_with_index.map do |url, index| Thread.new do results[index] = begin Net::HTTP.get_response(URI(url)).code rescue StandardError => error error.class.name end end end threads.each(&:join) urls.each_with_index { |url, index| puts "#{results[index]} #{url}" } failures = results.count { |code| code != "200" } exit(failures.zero? ? 0 : 1)

The rescue turns a dead host into a result instead of a crash: a SocketError is printed like any other answer, and the exit code still counts it as a failure.

A good URL and a bad one

The project uses a local server so the test does not depend on the internet. Two routes, one real and one missing:

require "webrick" server = WEBrick::HTTPServer.new(Port: 8000) server.mount_proc("/ok") { |_req, res| res.body = "ok" } server.start

urls.txt

http://localhost:8000/ok
http://localhost:8000/missing

ruby bin/linkcheck urls.txt

200 http://localhost:8000/ok
404 http://localhost:8000/missing

The exit code

The report is for a person; the exit code is for the pipeline. With one broken URL the command exits 1:

ruby bin/linkcheck urls.txt; echo $?

200 http://localhost:8000/ok
404 http://localhost:8000/missing
1

Point it at two working URLs and the same command exits zero, which is the only difference a pipeline sees:

ruby bin/linkcheck good.txt; echo $?

200 http://localhost:8000/ok
200 http://localhost:8000/ok
0

What to watch for

Do not start one thread per URL for a list of a thousand. A small pool of workers pulling from a Queue keeps the load on the target host sane and the memory flat; the pattern is in lesson 129.

A checker that follows redirects can loop. Net::HTTP does not follow them by default, so a 301 in the report is a real answer, and a link that moved is worth seeing once.

Next: Project: CSV to JSON converter

Article author: Arthur Isaev