Project: CSV to JSON converter

The converter

The third project is the data step of a pipeline: a spreadsheet export arrives as CSV, an API wants JSON. The converter reads the file, checks that the columns it promises are there, and writes pretty JSON so a person can read the output.

lib/table.rb

require "csv" class Table REQUIRED = ["name", "age"].freeze def self.read(path) rows = CSV.read(path, headers: true) missing = REQUIRED - rows.headers raise ArgumentError, "missing column: #{missing.first}" unless missing.empty? rows.map(&:to_h) end end

bin/csv2json

#!/usr/bin/env ruby require "json" require_relative "../lib/table" begin rows = Table.read(ARGV[0]) rescue ArgumentError => error warn error.message exit 1 end File.write(ARGV[1], JSON.pretty_generate(rows)) puts "#{rows.length} rows written to #{ARGV[1]}"

The schema check is the part that earns its keep. Without it the converter happily writes a file with name and no age, and the failure moves to whatever reads the JSON, far from the bad CSV.

Convert a file

in.csv

name,age
Ada,36
Bob,28

ruby bin/csv2json in.csv out.json

2 rows written to out.json

cat out.json

[ { "name": "Ada", "age": "36" }, { "name": "Bob", "age": "28" } ]

Every value arrived as text, because CSV has no types. That is the same lesson as the converter in lesson 42: pass converters: :numeric when a column must be a number, and document which columns are which.

When a column is missing

A file with name and city is not the shape the converter promises, and it says so on stderr with exit 1:

ruby bin/csv2json bad.csv bad.json; echo $?

missing column: age
1

No output file is written, so a pipeline that runs the converter and then uploads the result fails at the first step, which is the order that wastes the least work.

The test

require "minitest/autorun" require_relative "../lib/table" class TableTest < Minitest::Test def test_reads_rows rows = Table.read("test/fixtures.csv") assert_equal 2, rows.length assert_equal "Ada", rows.first["name"] end def test_raises_on_a_missing_column error = assert_raises(ArgumentError) { Table.read("test/broken.csv") } assert_equal "missing column: age", error.message end end

ruby -Itest test/table_test.rb

Run options: --seed 4321

# Running:

..

Finished in 0.002225s, 898.9600 runs/s, 1797.9201 assertions/s.

2 runs, 4 assertions, 0 failures, 0 errors, 0 skips

What to watch for

CSV files arrive with a byte order mark, Windows line endings or a different delimiter. CSV.read(path, headers: true) handles the common case; a file from Excel may need col_sep: ";" or a bom|utf-8 encoding, and the error names the column it could not find.

For a file that does not fit in memory, replace CSV.read with CSV.foreach and build the JSON array by streaming. The schema check moves to the first row, which is enough.

Next: Project: package the CLI as a gem

Article author: Arthur Isaev