Scanning and parsing strings

scan and split

Two string methods cover the simple cases. scan returns every match, and with groups it returns one array per match; split with a regex cuts the string wherever the pattern matches:

ruby -e 'p "a=1 b=2".scan(/(\w+)=(\d+)/)'

[["a", "1"], ["b", "2"]]

ruby -e 'p "a=1; b=2".split(/;\s*/)'

["a=1", "b=2"]

scan keeps the order of the matches and ignores everything between them, which is why it is the method for pulling ids or numbers out of a page. split keeps the parts, so it is the method for a line with a separator.

StringScanner

When the pieces have an order, a regex per field is clearer than one large regex. strscan is in the standard library, and each scan consumes from the front:

ruby -e 'require "strscan"; s = StringScanner.new("a=1"); p s.scan(/\w+/), s.scan(/=/), s.scan(/\d+/)'

"a"
"="
"1"

Each call moves the scanner forward, so the next pattern starts where the last match ended. A pattern that does not match returns nil and moves nothing, which is what lets a loop test eos? and stop at the end.

A key-value parser

Six lines of scanner is a parser for a=1; b=2. The loop reads a key, the equals sign with its optional spaces, the value up to the separator, and then the separator itself:

require "strscan" def parse(str) s = StringScanner.new(str) out = {} until s.eos? key = s.scan(/\w+/) s.scan(/\s*=\s*/) value = s.scan(/[^;]+/) out[key] = value.strip s.scan(/;\s*/) end out end p parse("a=1; b=2") p parse("name = Ada; role = admin")

ruby kv.rb

{"a" => "1", "b" => "2"}
{"name" => "Ada", "role" => "admin"}

The scanner makes the grammar explicit: each line of the loop is one token. A regex would have to repeat the same alternation and would report a failure as one anonymous nil instead of the step that failed.

scan_until and rest

scan_until consumes everything up to and including a pattern, and rest shows what is left. The pair is the way to skip a header:

ruby -e 'require "strscan"; s = StringScanner.new("id=42&x=1"); p s.scan_until(/=/); p s.rest'

"id="
"42&x=1"

check and peek look without consuming, which is what a parser needs when two rules start with the same character. pos reports the offset, so an error can point at the character that broke the grammar.

What to watch for

A scanner loop that does not consume anything runs forever: if every scan returns nil, the position stays put. Advance with getch or raise on an unexpected character.

split with a regex that can match an empty string produces surprising empty fields, and scan with an empty-capable pattern matches between every character. Anchor the pattern to what it must consume.

Next: A text pipeline

Article author: Arthur Isaev