XML and HTML

REXML

rexml is a pure-Ruby XML parser in the standard library. It loads a document into an element tree, and root.text reads the text inside the root:

ruby -e 'require "rexml/document"; d = REXML::Document.new("<a>x</a>"); puts d.root.text'

A document with a declaration, comments and namespaces parses the same way. REXML is slower than the C parsers and memory-hungry on a large file; for a config or an API response of a few kilobytes it is the one with no install step.

Attributes and XPath

attributes reads an attribute and elements takes an XPath expression. The two cover most of what a small feed needs:

ruby -e 'require "rexml/document"; d = REXML::Document.new(%q{<user id="7"><name>Ada</name></user>}); p d.root.attributes["id"]; p d.elements["//name"].text'

"7" "Ada"

elements.each("//item") walks a list, and each_element_with_attribute filters by an attribute. An XPath that starts with a single slash is absolute; one that starts with .// searches below the current node, which is what a loop over many items wants.

HTML is not XML

REXML requires well-formed XML: every tag closed, every attribute quoted. A page from the web rarely is, and the parser says so:

ruby -e 'require "rexml/document"; REXML::Document.new("a ")'

.../rexml/parsers/baseparser.rb:456:in 'REXML::Parsers::BaseParser#pull_event': Missing end tag for 'br' (got 'p') (REXML::ParseException)

The line number follows the message. For HTML the answer is not to loosen the XML parser but to use one written for HTML, which knows the void elements and the optional closing tags.

When a parser gem is needed

nokogiri wraps libxml2 and a forgiving HTML parser, and it speaks CSS selectors as well as XPath. It builds a native extension, so these commands ran on the host in the ruby:3.4 image:

gem install nokogiri

1 gem installed

ruby -rnokogiri -e 'doc = Nokogiri::HTML("a "); p doc.at_css("p").text'

"a"

The parse repaired the unclosed br and the selector still found the text. The same library parses XML with Nokogiri::XML, and doc.errors lists what it had to repair — the detail that makes it usable on real pages.

A project that only reads a small, well-formed feed does not need the dependency. A scraper does, and the Gemfile says so.

What to watch for

An XML parser that expands entities is a security problem: a small document can define an entity that expands into gigabytes. REXML has limits, and Nokogiri::XML takes NONET to refuse network entities. Parse documents from strangers with the restrictive options.

text returns the text of one node; doc.text concatenates the whole tree, including whitespace between tags. Normalise the whitespace before comparing two documents.

Next: Configuration files

Article author: Arthur Isaev