firepear

I love testing. I like to know, to the greatest degree manageable, that the code I have written does what I think it does. I love testing in Perl because there's a selection of tools available which is broad and deep. I wish the implementors of other languages would take a look at the Test:: and Devel:: sections of the CPAN and steal some ideas, really.

Recently I found myself designing a message-passing framework, and I decided that I wanted to at least have Perl 5, ObjC, Python, and Javascript implementations. The arguable sanity and hubris of this aside, it brought up an interesting problem in the domain of testing.

Having been influenced by Perl 6's awesome collection of spectests, I decided that the right thing to do was draft a spec have a suite of tests to go along with it. But then I realized I'd have to rewrite the spectests for every implementation, which, aside from being more work, increased the risk of programming errors in the tests themselves. I realized that if there were a way to write spectests (or API tests, or any other black-box tests) just once, that would eliminate the risk of programming errors in the test suite and eliminate the chance of the spec being interpreted in multiple ways, as the spec test suite would be definitive.

So I wrote Catechesis.

The idea is simple: tests are written in a simple, declarative format. Each test has (at a minimum) one "send" and one "expect" directive, and the values of these directives are JSON-encoded data structures.

A driver program (supplied with Catechesis) reads the test scripts and spawns a shim program. The shim translates the "sent" directives into native calls to the implementation being tested, formats the results to match the "expect" directive, and sends it back to the driver. Each implementation needs its own shim, but they should be (comparitively) tiny, and don't need to know anything except how to talk to STDIN/STDOUT and how to speak JSON.

Et voila, fully decoupled spec/API testing.

Oh, and the whole thing generates TAP as output, so it should be easy to integrate it into the overall testing ecosystem.

You can read more at the project's homepage or on the CPAN. A very simple math API is included, along with a Perl 5 implementation and shim -- I'll be writing a Python version soon to fully illustrate the cross-language nature of things, but I wanted to get it out the door ASAP :)

P.S. Yes, as in "catechism". The call/response of the driver to the shim (to show understanding of the spec) inspired me.