Property-based testing in Elixir using PropEr

Writing PropEr tests in Elixir

To explain writing PropEr tests (named "properties") in Elixir, let’s take an example from PropEr testing. A property of a function named biggest/1 is that the returned value is equal to the biggest value in the passed list.

# test/prop_biggest.erl
-module(prop_biggest).
-include_lib("proper/include/proper.hrl").

prop_biggest() ->
    ?FORALL(List, non_empty(list(integer())),              #1
        begin
            biggest(List) =:= lists:last(lists:sort(List)) #2
        end).

?FORALL is a macro around proper:forall/2. It takes a variable name, a generator, and a property. In this case, the generator returns a random, non-empty list of integers and puts it in the List variable for every run.
The property itself is a function that uses the random variable to assert the result from biggest/1 matches the biggest value in the list.

Converting it to Elixir, this property looks a bit different.

# test/biggest_prop.exs
defmodule BiggestProp do
  import :proper #1
  import :proper_types #1

  def prop_biggest do
    forall(non_empty(list(integer())), fn(list) ->    #2
      biggest(list) == list |> Enum.sort |> List.last #3
    end)
  end

  # ...
end

The :proper and :proper_types modules are imported manually instead of including the proper.hrl header file, which is a mostly a list of imports itself.
Elixir doesn’t support Erlang’s macros, so :proper.forall/2 is called directly.
Although Erlang’s lists module would work, Enum.sort/1 and List.last/1 are used as trusted functions to test the implementation.

With a working implementation, and :proper included in the project’s dependencies, the property can be run in IEx by requiring the test file manually and running the property through :proper.quickcheck/1.

$ iex -S mix
iex(1)> Kernel.ParallelRequire.files(["test/biggest_prop.exs"])
[BiggestProp]
iex(2)> :proper.quickcheck(BiggestProp.prop_biggest())
....................................................................................................
OK: Passed 100 test(s).
true

rebar3_proper and mix_proper

Although that worked, it would be nice to have a command to quickly run all tests in a Mix project. For Erlang, there’s a library named rebar3_proper that does just that by adding the rebar3 proper command.

$ rebar3 proper
===> Testing prop_biggest:prop_biggest()
....................................................................................................
OK: Passed 100 test(s).
===>
1/1 properties passed

rebar3_proper runs tests by finding functions and modules with names prefixed with "prop_" and running them through proper:quickcheck/1. That would work in Elixir by adding a Mix task that uses the same approach.

defmodule Mix.Tasks.Proper do
  use Mix.Task

  def run([]) do
    "test/**/*_prop.exs"
    |> Path.wildcard
    |> Kernel.ParallelRequire.files # => [BiggestProp]
    # ...
  end

  # ...
end

Using test/*\*/\*_prop.exs as a wildcard pattern, the paths for all property-based testing files in the test directory are found. Being .exs files, they are not compiled, so they need to be required manually when they’re needed. Mix’s own test task uses Kernel.ParallelRequire.files/1-2, which takes a list of filenames, includes the modules in the files and returns the names of the newly included modules.

defmodule Mix.Tasks.Proper do
  use Mix.Task

  def run([]) do
    "test/**/*_prop.exs"
    |> Path.wildcard
    |> Kernel.ParallelRequire.files
    |> Enum.each(fn(module) ->
      module.__info__(:functions)
      |> Enum.filter(&property?/1)
      |> Enum.map(fn({name, 0}) ->
        :proper.quickcheck(apply(module, name, []))
      end)
    end)
  end

  # ...
end

Using the list of included modules, their property functions are found using the __info__/1 function. Functions without the proper name or an arity other than 0 get filtered out. The remaining functions in the list are called using :erlang.apply/3, and their results are passed to :proper.quickcheck/1, which runs the tests.

$ mix proper
....................................................................................................
OK: Passed 100 test(s).

That’s it. The proper Mix task finds property-based test files, includes them, and runs each of the properties in those files through PropEr. The code above is the basis of mix_proper, which can be used in your Elixir project by adding it as a dependency.