Testing code that's testing itself

It’s quite common for testing libraries (like RSpec and Bacon) to test themselves. That’s pretty cool, but it might get you into some problems. In this article I’ll try to explain one of those problems and give a suggestion that might help.

Instead of showing you a red dot and a failing test, RSpec won’t be able to run its own tests at all if you break RSpec::Core::Example.run. That shouldn’t be anything shocking, because that method is very important to be able to actually run an example.

The problem we have here is that code you’re working on is unstable and can’t be trusted, which means you can’t really use it to test anything either. That’s testing broken code with broken code and it can get very confusing very fast.

Here’s a more simple example. Let’s say we added a method called #one to Object, allowing us to check the numerical value of ‘one’ by just calling #one on any object:

class Object
  def one
    1
  end
end

We also have a method to check if something’s value is 1, and we put it conveniently in the same library:

class Object
  def one
    1
  end

  def is_one?
    self == 1
  end
end

Now, we can write a test that makes our library test its own method:

one.is_one?

Nice. Now the library is using its #is_one? method to test its #one method. There’s only one problem though. What if we dive in again and start hacking on our library? We could break #is_one? and end up with something like this:

class Object
  def one
    1
  end

  def is_one?
    self == 2 # oops!
  end
end

This would mean that our test for #one will fail, while it’s not broken at all. Instead, our test is broken (which happens to be in the library we’re currently testing).

A solution would be to let a stable version of our library test the current version. Let’s try that.

In our test, we require the library to be tested first and clone it into a Unstable namespace. After that, we require a stable version of our library:

# library_spec.rb

require 'library'

module Unstable
  Object = ::Object.clone
end

require 'stable_library'

The stable library has overwritten the Object in the global namespace, but not the Unstable::Object. Now we should call #one from the Unstable namespace instead of the global one (because Unstable::Object is the class we’re currently testing):

Unstable::Object.one.is_one? # => 1

Because Unstable::Object#one returns a stable Fixnum, our stable version of #is_one? is used. This makes sure our test actually runs, even if the current (unstable) version of our library is broken.

Now we have a setup that uses a stable version of a library that can find bugs in an unstable version of itself. Awesome.

What do you think? Be sure to let me know if you have any ideas to improve this approach.

Jeff Kreeftmeijer

Testing code that's testing itself