mix test.interactive is an interactive test runner for Elixir’s ExUnit. It automatically runs your tests using mix test whenever you save a change to your code or tests and allows you to interactively modify the arguments passed to mix test.

While I released the initial version of this project almost four years ago, I’ve never written about it or publicized it, other than a podcast appearance.

Background

While doing test-driven development in a number of languages, I’ve come to appreciate a workflow where my tests run automatically every time I save changes to my source code or tests. I even open-sourced an automatic test runner for Smalltalk when I was working in that language.

When I started working in Elixir, I quickly found and adopted Louis Pilfold’s mix test.watch. This was (and still is) a wonderful tool that largely gave me the workflow I’d become used to.

I later started working on some JavaScript projects where we used Jest.

One of Jest’s features is an interactive watch mode, where Jest stays running and re-runs tests on every change, much like mix test.watch. However, it also provides a number of commands that can interactively modify which tests will be run.

I found this feature of Jest very handy, and realized that I often wanted to change the arguments I was passing to mix test.watch. I’d then have to exit it and restart it with the new arguments. Or, I’d run mix test separately in another shell when I needed different arguments.

After I spent some time wishing quietly that someone would build a Jest-like interactive test runner based on mix test.watch, I realized that it probably wasn’t going to happen unless I did it.

So, back in late 2020/early 2021 I started experimenting with the idea. Once I had the basics working, I asked Louis if he’d like me to contribute this back to mix test.watch. He’s been busy building Gleam, so he suggested that I should keep our projects separate. I did just that, releasing mix test.interactive v1.0 in February, 2021.

The Basics

When using mix test.interactive, you’ll run it from a command line, optionally passing some arguments to it. It will stay running until you quit it (with the q command, Ctrl-D, or the ever-popular Ctrl-C Ctrl-C).

While running, it watches for changes to your source code and re-runs the tests when it detects a change. So far, this is the same as mix test.watch.

mix test.interative also allows you to enter commands that change what arguments will be passed to mix test.

Internally, mix test.interactive keeps track of a number of settings which are used to construct a command-line for mix test. These settings are updated every time an interactive-mode command is entered.

Examples

If my test suite is very fast, I’m usually fine with running all of my tests whenever anything changes. This is the default mode of mix test.interactive.

If I want even faster feedback, I might only want to run stale tests; that is, only tests affected by code/test changes made since the tests last passed. For this, I can use the s command to run only the stale tests, which runs mix test --stale under the hood.

In a larger application, I might only want to run tests matching a filename pattern. For example, if I’m working in my application’s Accounts context, I can use the p accounts command to run only those tests whose filenames contain the substring (pattern) accounts.

Perhaps I’ve just made a fundamental change to my application that caused a number of tests to fail. I’d like to keep re-running the failed tests until I can get them all passing again. For that, I can use the f command, which runs mix test --failed under the hood.

I can switch back to running all of the tests again with the a command.

Maybe I want to run a single test at a time, but change which test I’m focused on at any moment. I could add @tag :focus above the test I’m interested in and then use the o focus command, which will run mix test --only focus. I can then move the @tag :focus line to a different test to run that test instead.

If I’m making a number of changes and don’t want to run the tests while making them, I can turn watch mode off with the w command, make the changes, and then press Enter or turn watch mode back on (w command again) to run the tests.

Current Status

As I mentioned, I released the first version of mix test.interactive in February of 2021 and I’ve been enhancing it off and on since then. I use it on every Elixir project I work on. That doesn’t mean it’s perfect, but at this point, it’s got all of the features I can think to add to it.

It can manage the following mix test options via interactive-mode commands:

  • --failed (f command)
  • --exclude <tag> (x command)
  • --include <tag> (i command)
  • --max-failures <count> (m command)
  • --only <tag> (o command)
  • --repeat-until-failure <count> (r command) (Elixir 1.17.0 and later)
  • --seed <seed> (d command)
  • --stale (s command)
  • --trace (t command)
  • <filename> or <filename:line_no> arguments (p command)

I’m happy to consider additional feature requests or bug reports.

Wish List

There are two things I’d love to change in mix test.interactive, but haven’t been able to figure out how yet.

Immediate Response

Jest works by immediately processing keystrokes, so there is no need to press Enter after specifying a command.

As near as I can tell, Elixir (and, under the hood, Erlang) do not support immediate input mode out of the box (the equivalent of the C getc/getchar functions).

One suggestion I saw was to use an Elixir wrapper of the ncurses library, but I’m reluctant to add such a dependency because I’m not sure how well it would work on all platforms, including Windows.

If anyone knows how to do this, I’d love some pointers!

Tab-completion for p Command

I’d love to allow tab-completion of arguments to the p command.

From the documentation, I see that Erlang’s :io module has an expand_fun option that can be set that is supposed to be active when calling :io.get_line, but I’ve unable to make it work in mix test.interactive.

Again, any pointers on how to accomplish this would be most welcome!

Wrapping Up

If you think the mix test.interactive approach would work for you, I’d love it if you give this tool a try. It doesn’t interfere with anyone else’s workflow, so it is safe to add to any project for those that want to use it, while staying out of the way of anyone that doesn’t. Check it out on hex.pm!