Given the popularity of Joel Spolsky, or at least his blog, I feel like I might be going out on a limb here. But unless I’m missing something (OK I admit it, I did only watch the edited highlights), in his interview with Robert Scoble today, he points out that FogBugz was designed to make it “very difficult to get metrics”. He says that when they look at feature requests at Fog Creek, they think about features “anthropologically”, and prefer to leave those features out if they can be misused by lazy managers and therefore lead to developers working around the bug tracking system.

Isn’t Joel looking at the wrong end of the problem? Sure, leave out the features that encourage misuse, but rather than avoiding metrics altogether, why not put more effort into trying to find out what metrics could be used to good effect to improve the process, and promote them like crazy? “The managers” seem to be positioned as the bad guys in this scenario; personally, I wouldn’t make that distinction - we’re all members of the same team, and should have the same objectives.

I just re-read an article that I came across last month on the subject of The State of Software Quality. It’s an odd piece, to me, because it starts with a lot of doom and gloom about the state of software quality, but finishes up with a ray of sunshine from The Standish Group, whose latest survey indicates that finally, things seem to be improving.

But to me, the most telling quote in the piece comes from Watts S. Humphrey, who founded the Software Process Program of the Software Engineering Institute at Carnegie Mellon University, and was the director of programming at IBM.

Part of the problem, said Humphrey, is that many organizations are focusing on tools. “People think if you want higher-quality stuff, you’ve got to get better tools,” he said.

Rather, the biggest impact on quality are the people using the tools, and the people they report to, Humphrey and other experts interviewed for this story said. “To get a quality product you need two things: programmers personally committed to producing high-quality stuff and managers and customers that demand high-quality work,” Humphrey said.

Amen to that! When you think about it, there are many, many tools available that can be used to improve code quality. Many of them are free. What we need is a change in the mindset of practitioners, managers and customers, and some innovative thinking in terms of how to apply quality tools and techniques to an existing and entrenched development process.

Our experience in working with clients is that many shops are not using development quality metrics simply because they don’t know where to start. It’s the old problem of eating an elephant - the task seems so huge, you don’t know where to start, so you never get started.

Pavel Senko posted a nice piece last month suggesting 6 quality metrics that could easily be collected, that would start to give you a picture of the state of the quality of your code base:

1) Cumulative Metrics - even for something as simple as incoming versus resolved defects, this gives you a quick view of whether things are getting better or worse over time.
2) Defects Backlog - sort of a product of (1) - this gives you a view of whether unresolved backlogs are piling up.
3) Release to Release Comparison - by comparing the performance of one release against the previous one, you can see whether quality is improving or not.
4) Mean Time To Resolution - perhaps a bit more controversial, this one measures how quickly defects are being resolved.
5) Running Total Metrics - in his example, Pavel measures cumulative 2 months of incoming defects from customers.
6) Phase Containment Metrics - keeping track of how many defects were found in each phase can give an indication of the cost to the business of defects.

Our experience is that very few development shops are collecting quality metrics of any sort. If that describes you, and you are looking for a place to start, Pavel’s post may give you some pointers.

Back in January, Jason Barile did a nice job of trying to define confidence when it comes to judging whether your product is ready to ship. He contrasts “running tests and reporting a high pass rate” against “standing up in front of your senior management team” … “and stating that your product is ready to ship”.

Jason talks about how using a variety of quality metrics can help raise your confidence level in that situation.

Trouble is, we really don’t know what metrics are the best predictors of “quality”. If we did, life would be a lot easier…

Software quality tools vendors have a vested interest in perpetuating the idea that most software is written to low standards and the industry needs a shake up (or to buy our tools) to get itself back in shape. I was put in mind of this reading a recent Borland press release, which includes the quote “The consequences of poor software quality are well known, yet quality is still often treated as an afterthought - something addressed late in the development lifecycle”. Doesn’t that quote sound familar?

So how bad are things, really? Here at Enerjy, we’ve been spending quite a bit of time recently trying to take an objective view of the quality of open source software projects, and I have to report that, at first glance, anyway, quality levels in the open source world, at least those projects that are popular and well used, are pretty high. Many commercial projects we see fall far short of the standards set by open source projects. Why would that be?

Well, it could be that the very transparency of the open source code encourages best practices (people are going to see my code, so I want to make sure it’s good). Or, it could be that most of the open source projects we looked at used static analysis and unit test coverage tools (good, open source tools are available for both those areas). Or, it could be that lack of pressure from “the business” meant that the developers weren’t being forced to cut corners to get the release out the door before it was really ready.

Whatever the reason, what’s needed, it seems to me, is some agreement in the industry on what metrics should be used to measure the quality of a piece of software, and what standards we should apply before we can truly say that an application is ready for prime time.

Hi - I’m Nigel Cheshire, CEO of Enerjy Software, and I started this blog because I think we have some interesting things to say about software quality. We also want to hear what others have to say about it, and I’m hoping that eventually, more people will enter the dialog. Time will tell!

The purpose of this blog is not to talk about Enerjy. If you want to find out about us (and of course I hope that you do), you can do that elsewhere. The purpose of this blog is to give us an outlet to talk about the things we care about, maybe to explain a little bit about why we do what we do, and especially why we do it the way that we do it. You’ll see contributions on here from a number of the folks here at Enerjy (at least I hope you will).

Sure, there are plenty of other people out there already talking about this subject, but given that software quality is what we live for, we thought it was time for us to jump in.