Property-based testing in Elixir using PropEr

By on

While reading Fred Hébert’s PropEr testing, on property-based testing in Erlang, I set out write PropEr tests in Elixir, and run them with a Mix task.

mix_proper running property-based tests in an Elixir project

Writing PropEr tests in Elixir

To explain writing PropEr tests (named “properties”) in Elixir, let’s take an example from PropEr testing. A property of a function named biggest/1 is that the returned value is equal to the biggest value in the passed list.

# test/prop_biggest.erl
-module(prop_biggest).
-include_lib("proper/include/proper.hrl").

prop_biggest() ->
    ?FORALL(List, non_empty(list(integer())),              # ➊
        begin
            biggest(List) =:= lists:last(lists:sort(List)) # ➋
        end).
  1. ?FORALL is a macro around proper:forall/2. It takes a variable name, a generator, and a property. In this case, the generator returns a random, non-empty list of integers and puts it in the List variable for every run.

  2. The property itself is a function that uses the random variable to assert the result from biggest/1 matches the biggest value in the list.

Converting it to Elixir, this property looks a bit different.

# test/biggest_prop.exs
defmodule BiggestProp do
  # ➊
  import :proper
  import :proper_types

  def prop_biggest do
    forall(non_empty(list(integer())), fn(list) ->    # ➋
      biggest(list) == list |> Enum.sort |> List.last # ➌
    end)
  end

  # ...
end
  1. The BiggestProp module imports :proper and :proper_types manually instead of including the proper.hrl header file.

  2. Elixir doesn’t support Erlang’s macros, so it calls :proper.forall/2 directly.

  3. Although Erlang’s lists module would work, it uses Enum.sort/1 and List.last/1 as trusted functions to test the implementation.

With a working implementation, and :proper included in the project’s dependencies, the property runs in IEx by requiring the test file manually and running the property through :proper.quickcheck/1.

$ iex -S mix
iex(1)> Kernel.ParallelRequire.files(["test/biggest_prop.exs"])
[BiggestProp]
iex(2)> :proper.quickcheck(BiggestProp.prop_biggest())
....................................................................................................
OK: Passed 100 test(s).
true

rebar3_proper and mix_proper

Although that worked, it would be nice to have a command to run all tests in a Mix project. For Erlang, there’s a library named rebar3_proper that does just that by adding the rebar3 proper command.

$ rebar3 proper
===> Testing prop_biggest:prop_biggest()
....................................................................................................
OK: Passed 100 test(s).
===>
1/1 properties passed

rebar3_proper runs tests by finding functions and modules with names prefixed with "prop_" and running them through proper:quickcheck/1. That would work in Elixir by adding a Mix task that uses the same approach.

defmodule Mix.Tasks.Proper do
  use Mix.Task

  def run([]) do
    "test/**/*_prop.exs"
    |> Path.wildcard
    |> Kernel.ParallelRequire.files # => [BiggestProp]
    # ...
  end

  # ...
end

Using “test/**/*_prop.exs” as a wildcard pattern, the task finds the paths for all property-based testing files in the test directory. Being .exs files, they are not compiled, so they’re required manually when they’re needed. Mix’s own test task uses Kernel.ParallelRequire.files/1-2, which takes a list of filenames, includes the modules in the files and returns the names of the newly included modules.

defmodule Mix.Tasks.Proper do
  use Mix.Task

  def run([]) do
    "test/**/*_prop.exs"
    |> Path.wildcard
    |> Kernel.ParallelRequire.files
    |> Enum.each(fn(module) ->
      module.__info__(:functions)
      |> Enum.filter(&property?/1)
      |> Enum.map(fn({name, 0}) ->
        :proper.quickcheck(apply(module, name, []))
      end)
    end)
  end

  # ...
end

The task uses the list of included modules to find their properties using the __info__/1 function. Functions without the proper name or an arity other than 0 get filtered out. It then calls the remaining functions in the list using :erlang.apply/3, and passes their results to :proper.quickcheck/1 to run the tests.

$ mix proper
....................................................................................................
OK: Passed 100 test(s).

That’s it. The proper Mix task finds property-based test files, includes them, and runs each of the properties in those files through PropEr. The code above is the basis of mix_proper, which can be used in your Elixir project by adding it as a dependency.