A REST client that survives
The client
A request that fails once is normal: a connection reset, a 502 from a proxy, a timeout. The client below retries with a doubling wait, stops at a retry limit, and stops earlier when the total budget is gone:
require "net/http"
require "json"
class Client
def initialize(url, retries: 3, budget: 5)
@uri = URI(url)
@retries = retries
@budget = budget
end
def get
deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @budget
attempt = 0
begin
attempt += 1
res = Net::HTTP.start(@uri.host, @uri.port, use_ssl: @uri.scheme == "https",
open_timeout: 2, read_timeout: 2) do |http|
http.get(@uri.request_uri)
end
raise "HTTP #{res.code}" unless res.is_a?(Net::HTTPSuccess)
JSON.parse(res.body)
rescue StandardError => e
left = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
raise if attempt > @retries || left <= 0
wait = 0.2 * (2 ** (attempt - 1))
puts "attempt #{attempt} failed (#{e.message}), retrying in #{wait}s"
sleep wait
retry
end
end
end
data = Client.new(ARGV.fetch(0)).get
puts "ok: #{data["url"]}"
The pieces that matter: the deadline is computed once with the monotonic clock, the retry condition tests both the count and the clock, and the waiting time doubles so a struggling server is not hit in a tight loop.
A real request
Against a service that answers on the first try, the retry code is invisible and the JSON has the shape the endpoint documents:
ruby client.rb https://httpbin.org/get
ok: https://httpbin.org/get
Retries against a flaky server
To see the retries, a local server answers 500 twice and 200 on the third request. The client logs each attempt:
require "webrick"
count = 0
server = WEBrick::HTTPServer.new(Port: 9292, Logger: WEBrick::Log.new(File::NULL), AccessLog: [])
server.mount_proc("/flaky") do |_req, res|
count += 1
if count < 3
res.status = 500
res.body = "not yet"
else
res.status = 200
res.body = '{"url":"/flaky"}'
end
end
trap("INT") { server.shutdown }
server.start
ruby flaky.rb &
ruby client.rb http://127.0.0.1:9292/flaky
attempt 1 failed (HTTP 500), retrying in 0.2s attempt 2 failed (HTTP 500), retrying in 0.4s ok: /flaky
The third attempt succeeds, and the caller sees one value. The waits add 0.6 seconds, which is the price of not failing the whole job for a transient error.
The rules that keep it safe
- Retry only idempotent requests. A GET, a PUT and a DELETE can be repeated; a POST that creates a row may create two. A retried POST needs an idempotency key the server honours.
- Retry the errors that go away. A 500, a 502, a timeout and a reset are worth another try; a 400 or a 401 will fail the same way forever.
- Bound both the count and the time. Three attempts with a growing wait is five seconds of a user's patience; a deadline stops the fourth.
- Log each attempt. The line above is what makes a slow request explainable after the fact.
What to watch for
rescue StandardError also catches the JSON parse
error, so a malformed body is retried as if the network failed. Parse
after the success check and let a parse error be a parse error, or rescue
the network classes by name.
A retry inside a request that already has a retry (a proxy, a load
balancer, a framework middleware) multiplies. Know how many layers retry,
and give the outer one the larger budget.
Next: Rack: what a Ruby web app is
Article author: Arthur Isaev