When we’re writing automated tests for our code, it’s easy to accidentally write a test that says exactly the same thing as the code does. This kind of test is a “tautological test”, and it’s important to watch out for them.

I’ve used the term “tautological test” for a long time without thinking about it until I saw a tweet by Aaron Patterson that made me realize that many people probably aren’t familiar with the term.

What is a Tautological Test?

According to Google, a tautology is “a statement that is true by necessity or by virtue of its logical form.”

The most basic form of tautological test is when the test and the code use the same formula. For example:

Basic Tautology
def sum(a, b)

  a + b

end

 
def test_sum

  assert_equal 5 + 3, sum(5, 3)

end

On the surface, this looks like an OK test. We make it really clear how the inputs map to the outputs. There are no magic numbers.

But the test and the code use exactly the same algorithm to compute the expected and actual results, so what are we actually testing?

In this case, it would be better to do the calculation in our head and write the answer into the test:

Fixing the Test
def test_sum

  assert_equal 8, sum(5, 3)

end

Most tautological tests are not that simple to see.

Chris Oldwood gives some nice examples in Tautologies in Tests, but there are subtleties to consider.

Subtleties

In one of his examples, Chris Oldwood shows how a test and the code under test might depend on the same underlying helper function, FormatTime. He shows that, if FormatTime was implemented to just return null, the test will still pass. That’s not good, and it’s a potential indicator of a tautological test.

Chris’ first solution is to duplicate the work of FormatTime in the test as a cross-check against the code. That way, if FormatTime changes in a dangerous way, our tests will break. This might be OK, but as Chris points out, it couples our code to the implementation of FormatTime, leading to brittle tests that can break for reasons unrelated to the code change being made.

As Chris says, we need to dig deeper.

In our current context, what is important? What do we care about? What are we trying to communicate to future readers of our code? Is there something about the formatted time that is important here?

If so, then by all means we should include something in our assertion about that, but make the assertion as loose as possible to accomplish our goal.

But if all we need is to ensure that the formatted time appears and we don’t need to care about the actual format, then the test is fine as it stands. But then, how do we ensure that someone doesn’t break FormatTime?

We need to write tests for it directly.

In my tests, if I’m going to have the test code and system under test rely on the same helper, I always make sure that the helper is well-tested on its own.

I tend to build up the higher levels of abstraction using lower-level well-tested building blocks.

I illustrate this approach in the context of testing Redux reducers in the Testing section of Encapsulating the Redux State Tree.

A Word About Test Doubles

If you search for “tautological unit tests”, you’ll come up with a number of articles that specifically refer to mocks and stubs being the source of many tautological tests.

For example, Fabio Pereira shows a really nice example in TTDD - Tautological Test Driven Development (Anti Pattern).

There’s also codertom’s How mocking can be a recipe for disaster.

Or you can read the much more absolute opinion of Mark Sands, Mocking is Tautological.

All of these articles make good points, and it is definitely true that test doubles (mocks and stubs) can be abused in ways that result in fragile test suites that cost more than they save.

I’ve written quite a bit about this, as has Justin Searls. So rather that just repeating all of the content here, I’ll refer you to that other writing.

In general, though, I find that when I think about the messages between objects and make a distinction between collaborators and helpers (using doubles for the former, but not the latter), I don’t often run into the kinds of problems outlined in these posts.

Here are some references if you want to dig deeper into this topic:

Avoiding Tautological Tests

The first step in avoiding tautological tests is to avoid tautological tests.

But really, the first step in avoiding tautological tests is to be aware of them so you can watch out for them.

I find that focusing on the externally-visible behavior of the code I want to write really helps. This helps get my mind out of the implementation details, which is where a lot of tautological tests come from.

We talked above about duplicating some code as a cross-check. Sometimes this is a good idea, but often there is a better way.

codertom gives a nice list of advice:

  • Never calculate an expected value to check against within your test.

  • Logic driving the assertions in your test code is always a smell.

  • Always work from real examples whenever you can.

  • Never write test code that assumes it knows how the method under test should be implemented.

These are excellent guidelines and a good place to start. Even though these guidelines are written as extremes (“always” and “never”), use your best judgement.