My initial reaction to Andrew Binstock’s post on Effectiveness of Pair-wise Tests, was: “Excellent! A way of getting a better return on developers’ testing time. Surely that’s what every manager wants!”

Resource allocation for testing legacy code (I steal Michael Feathers’ definition here – legacy code is any code that hasn’t been unit tested) is one of the biggest factors when considering code quality improvement initiatives. It’s the same old story: the development team understands that this should be done and [most] understand the benefits it will bring, however, the business demands that applications get to the customer on time regardless (and I do understand the business reasons for this).

In one example, Binstock gives results from BJ Rollinson’s testing of attrib.exe (which takes a path + 6 optional args) using Pairwise testing as:

Minimal: 9 tests, 74% code coverage, 358 code blocks covered.
Pairwise: 13 tests, 77% code coverage, 370 code blocks covered
Maximal: 972 tests, 77% code coverage, 370 code blocks covered

77% code coverage represents a very good return for the work involved, but “hold on”, I thought, “this means that almost a quarter of this code is still untested”. I haven’t seen the details, but I would be willing to bet that part of the untested code is exception handling code (please correct me if I’m mistaken, Mr. Rollinson).

There is no doubt that new methods to get best practices such as unit testing into the mainstream development lifecycle (as efficiently as possible) represent much needed and welcome progress. I also believe that setting a procedure for unit testing legacy code is even more vital (especially in the short term). Not only can lower hanging fruit be plucked to make commonly used parts of the application more robust, but also one of the most common security violations (the dreaded output to the screen of a stack trace when an error occurs) can easily be eliminated.

In almost every application I have seen unit tested, this lack of attention to exception handlers is a common oversight. Every developer knows that users have an uncanny knack of using our applications in ways that we never intended. When that happens, we need to consider what they see. If you are working for a financial company on some sort of web banking/payments application, what information could your users get access to?