Getting Testy: Learning

This post is part of an ongoing series about Unit Testing.

Introduction

In the previous post, we talked about writing good acceptance or story tests. Now that that outer layer of tests is in place, it’s time to start developing the application itself.

As you may recall, we’re building a simple application that can do two things:

1. Convert an amount of money from one currency to another.

2. Show a list of all currencies supported by the application.

We still haven’t decided what kind of application this should be, but we’re at the point where we need to make that decision.

Let’s start with a command-line application just to get things running. We may convert it into a web application later, or maybe just add a web front-end and allow it to be used either way.

What Now?

With any new project, there are likely some things we’re not sure about. For our command-line application, we might need to learn about how to handle the command-line options. For a web application, we might want to try out Sinatra but aren’t quite sure how it works. We also might want to experiment with a few different APIs for getting at the currency data we need for the application.

One way to resolve these uncertainties is to open up irb and start poking around. But another way is to write learning tests, an idea that Kent Beck credits to Jim Newkirk and Laurent Bossavit in his Test-Driven Development By Example book. James Grenning has also written a nice blog post on the subject.

We can use our test framework to help us explore the landscape and to capture what we learn along the way.

Learning OptionParser

Let’s look at how we might apply learning tests to Ruby’s OptionParser. We can first review the documentation to get a general sense of how to use the class, but we’ll really want to experiment with a few ideas.

Simple Help Option

Let’s figure out how to get a basic help option to work with OptionParser. We’ll use Minitest for this post.

From reading the documentation, it looks like we need to implement the help option if we want one. The implementation should print some kind of usage string on $stdout and then exit. We can use Minitest’s assert_output for that.

There’s a good example of how to implement the help option in the documentation, but let’s first make sure we can create a basic OptionParser and see what it does. We don’t know the exact format of the output, so we can just expect an empty string and the assertion failure will tell us.

All of the examples in the documentation use the parse! method, and it looks like we can pass in an array of strings in order to simulate the command-line arguments passed to the program. That seems really handy for tests, so let’s do that.

class OptionsTest < Minitest::Test
  def test_help
    parser = OptionParser.new
    assert_output("", "") do
      parser.parse!(%w[-h])
    end
  end
end

assert_output takes two positional parameters and a block. The block is run, and any output to $stdout and $stderr is captured and compared against the two parameters. If the output matches, the test passes; if not, it fails.

When we run these tests, they just abort prematurely. After a little bit of digging, we figure out that OptionParser is exiting for us. That’s not too helpful when you want to run an entire test suite.

A bit of searching tells us that we can stop the exit from happening by rescuing the SystemExit exception. Minitest has assert_raises that helps with this:

def test_help
  parser = OptionParser.new
  assert_output("", "") do
    assert_raises(SystemExit) do
      parser.parse!(%w[-h])
    end
  end
end

Now we get a test failure, because the output doesn’t match the empty string we passed in. Given that we haven’t implemented a help option yet, we expected an error message to that effect. Instead, we got actual output showing a usage message. OptionParser must have a pre-built implementation of the help option, though the documentation doesn’t really mention that.

The output contains a program name we don’t recognize. When using Rake’s test task, the program name is rake_test_loader; when running from RubyMine, it’s tunit_or_minitest_in_folder_runner. We don’t really want to couple our tests to one way of running them, so we go back to the documentation. It turns out that we can tell the OptionParser what program name to use.

def test_help
  parser = OptionParser.new do |p|
    p.program_name = "PROGRAM"
  end
 
  assert_output("Usage: PROGRAM [options]\n", "") do
    assert_raises(SystemExit) do
      parser.parse!(%w[-h])
    end
  end
end

I’m using a simple, obvious string (PROGRAM) as the program name so that it shows up clearly in the assertion failures.

Having a fixed program name allows us to fill in the proper expected output string, and our first learning test is now passing.

Version Option

Every command-line application should be able to report its version. We’ll write that test next. First, let’s pull the OptionParser up into a setup method so we don’t have to keep redefining it in every test.

class OptionsTest < Minitest::Test
  def setup
    super
    @parser = OptionParser.new do |parser|
      parser.program_name = "PROGRAM"
    end
  end
 
  def test_help
    assert_output("Usage: PROGRAM [options]\n", "") do
      assert_raises(SystemExit) do
        parser.parse!(%w[-h])
      end
    end
  end
 
  private
 
  attr_reader :parser
end

I’ve used a private attr_reader for the parser instance variable so that I can use barewords. It doesn’t strictly need to be private, but I’ve gotten in that habit to keep from exposing internal state from my classes. Admittedly this is less important in a test class.

Now we can write a test for the version option.

def test_version
  assert_output("", "") do
    parser.parse!(%w[--version])
  end
end

Once again, we just pass in empty strings until we know the exact output format. And once again, the tests exit prematurely. OptionParser must be exiting for us here, too. Maybe it also implements the version option for us? Let’s use the same trick we did for the help option.

def test_version
  assert_output("", "") do
    assert_raises(SystemExit) do
      parser.parse!(%w[--version])
    end
  end
end

Once again, we get the test failure we expect. And, as we suspected, there seems to be a built-in implementation of the version option, but it’s saying version unknown. How do we tell it what version to report? According to the documentation, there’s a version attribute we can set.

def setup
  super
  @parser = OptionParser.new do |parser|
    parser.program_name = "PROGRAM"
    parser.version = "VERSION"
  end
end
 
def test_version
  assert_output("PROGRAM VERSION\n", "") do
    assert_raises(SystemExit) do
      parser.parse!(%w[--version])
    end
  end
end

Again, we use an obvious literal string for the version and add the actual expected output. We have another passing test.

A Flag Option

Next, we’re curious about how to pass in a flag to enable an option. Our application will need a way of knowing when to output a list of currencies. One way we might do that is to allow a -l or --list option.

From the examples in the documentation, it seems like the thing to do is to define a Hash, Struct, or OpenStruct to hold the options, and then have the OptionParser populate the options based on the command-line flags. Let’s try that.

def setup
  super
  @options = OpenStruct.new
  @parser = OptionParser.new do |parser|
    parser.program_name = "PROGRAM"
    parser.version = "VERSION"
 
    parser.on("--list", "Show currency list") do |flag|
      options.show_list = flag
    end
  end
end
 
def test_list_flag
  parser.parse!(%w[--list])
  assert(options.show_list)
end
 
attr_reader :options, :parser

The new test passes as soon as we add the list option to the parser in setup, but test_help fails because the actual output now includes a description of the new list option. Let’s adjust for that.

We could write a complicated regex to match the entire help output string, but instead let’s just test parts of the output. We can use Minitest’s capture_io and assert_match features for that.

def test_help
  out, err = capture_io do
    assert_raises(SystemExit) do
      parser.parse!(%w[-h])
    end
  end
 
  assert_empty(err)
 
  assert_match(/^Usage: PROGRAM \[options\]$/, out, "output should contain usage message")
  assert_match(/^\s*--list\s*Show currency list\s*$/, out, "output should describe --list option")
end

A Short Flag Option

We want to be able to use the shorter -l option for listing currencies, so let’s test that as well.

def test_short_list_flag
  parser.parse!(%w[-l])
  assert(options.show_list)
end

Surprisingly this test passes right away, even though we haven’t specified a short option for --list. OptionParser must be helping again. I’d feel better if we were explicit about the short option, so let’s add it.

def setup
  super
  @options = OpenStruct.new
  @parser = OptionParser.new do |parser|
    parser.program_name = "PROGRAM"
    parser.version = "VERSION"
 
    parser.on("-l", "--list", "Show currency list") do |flag|
      options.show_list = flag
    end
  end
end

The new test is still passing, but our help test fails again. We need to tweak it a bit for the new help output. The help test seems pretty brittle; we’ll have to think about that when we’re done learning about OptionParser.

def test_help
  out, err = capture_io do
    assert_raises(SystemExit) do
      parser.parse!(%w[-h])
    end
  end
 
  assert_empty(err)
 
  assert_match(/^Usage: PROGRAM \[options\]$/, out, "output should contain usage message")
  assert_match(/^\s*-l, --list\s*Show currency list\s*$/, out, "output should describe --list option")
end

Other Options

In order to convert currencies with our command-line application, we might just pass in the amount and source and target currencies without any option flags. Something like currencyfx 100 USD EUR. Let’s see how OptionParser handles that.

We’ve been using the parse! method all along, since that’s what the examples use. The documentation says that parse! is the “[s]ame as parse, but removes switches destructively. Non-option arguments remain in argv.”

def test_non_option_args_left_behind
  args = %w[-l FOO BAR BAZ]
  parser.parse!(args)
  assert_equal(%w[FOO BAR BAZ], args)
end

That test passes right away, so it looks like we understood the documentation correctly.

Conclusion

We’ve now explored all of the features of OptionParser that we think we’ll need for the stories we’re working on.

We learned quite a bit about how OptionParser works along the way, including some surprising built-in behavior that isn’t documented. This would be an excellent time to file a documentation improvement PR. I’ll leave that as an exercise for the reader for now.

We can keep these tests as a record of our learning. Or we can use them as a reference for our actual implementation, then delete them once we’re testing our real code. We’ve even got a basic implementation of the OptionParser we’ll need in our real code.

The next time you need to work with an unfamiliar gem or library, try writing learning tests instead of just poking around in irb.