Testing the tester

30 August 2006

I recently wrote about the assert_cookie custom test assertion plugin I put together. When I started writing the assertion, it was just a helper method to wrap up the funky logic for testing stuff about a cookie. I wasn't thinking about tests. And my first tries were definitely not test-driven.

When I was wrapping it up as a plugin, this started bugging me a lot. Especially because my first tries at the assertion method weren't very good. (Ugh! svn diff -r59:72 http://svn.planetargon.org/rails/plugins/assert\_cookie/lib/assert\_cookie.rb) But the idea of testing it was confounding. How do you think about testing the tester?

The first thing I had to get straight was what I expected. It basically reduced to expecting the assertion to pass or fail. When an assertion fails, it raises the AssertionFailedError exception. Excellent, now I was making progress.

I started writing tests like:

1 def test_assertion_cookie_is_secure_should_pass_when_cookie_is_secure
2   assert_nothing_raised { assert_cookie :secret, :secure => true }
3 end
4 
5 def test_assertion_cookie_is_secure_should_fail_when_cookie_is_not_secure
6   assert_raise(AssertionFailedError) { assert_cookie :open_secret, 
7       :secure => true }
8 end

and notices a lot of repetition of assert\_raise(AssertionFailedError) and assert\_nothing\_raised. It was ugly to read. I decided to write a couple helper assertions:

 1 def assert_pass
 2   assert_block("assert_pass must be called with a block.") { block_given? }
 3   yield
 4 end
 5 
 6 def assert_fail
 7   assert_block("assert_fail must be called with a block.") { block_given? }
 8   yield
 9 rescue AssertionFailedError
10 end

The hard part again was figuring out what I was expecting. My first try was to use assert\_raise and assert\_nothing\_raised as before. But those failed miserable because they call yield. I lamented to Andrew that I wished yield would work like raise. After thinking a bit more, I realized I only cared whether an AssertionFailedError exception was raised or not. The previous tests now look like this:

1 def test_assertion_cookie_is_secure_should_pass_when_cookie_is_secure
2   assert_pass { assert_cookie :secret, :secure => true }
3 end
4 
5 def test_assertion_cookie_is_secure_should_fail_when_cookie_is_not_secure
6   assert_fail { assert_cookie :open_secret, :secure => true }
7 end

There is one last thing that bugs me. I wanted the tests to run outside of Rails but function correctly in Rails. Rails adds a method to Test::Unit::Assertions called clean\_backtrace. I couldn't figure out how to get that required correctly, so I created a stub in test\_helper.rb. Can anyone help me fix this?

Install the plugin with script/plugin install http://svn.planetargon.org/rails/plugins/assert_cookie. Navigate to the vendor/plugins/assert_cookie directory, type rake and watch the tests run. :)