As September draws to a close, and we start to think of pumpkins and apple cider, this week’s Glitch Watch mailbag put me more in mind of sandy beaches and waving palms.

First, from Christopher Columbus’ former private family estate Jamaica, comes the news that almost 29,000 incorrect income tax payment reminders were sent out, owing to a software error. A spokesperson for the Jamaican Tax Administration Services Department (TASD) explained that “The coding was set incorrectly and picked up information that should not have been picked up.” While you might expect them to send out corrected statements to all 29,000 recipients, in fact the TASD merely agreed to make statements available “on request.” I hope no-one gets confused about their liability; apparently Jamaican outstanding tax liabilities attract a penalty interest rate of 40%!

Meanwhile, in the neighboring Cayman Islands, a government watchdog is investigating whether Cayman Airways was guilty of price gouging for travelers attempting to evacuate ahead of Hurricane Dean in August. Cayman Airways CEO Patrick Strasburger claimed that the inflated prices were caused by a software glitch that temporarily caused ticket prices to go higher right before Hurricane Dean struck the island. Quite a coincidence!

Finally, a story that comes not from the sunny Caribbean, but the almost-as-exotic Saginaw, MI. A computer glitch at Saginaw’s Wendler Arena erroneously shut down one of the compressors that makes ice for the skating surface, and then the second compressor found it was working too hard and shut itself down for fear of overheating. I’m guessing that, for much of the year, Saginaw is not as hot as Jamaica. But it was hot enough last Saturday night, and the result, when the team showed up to practice on Monday, was a big puddle of water mixed with the paint used for the lines and logos set in the ice. So, not only did they have to re-freeze the water, but they had to repaint lines and logos as well. The work was expected to be finished by the end of the week.

Michael Bolton has been contemplating the question “What are the most useful metrics for code quality?”

Last month, I posted on some ideas put forth by analytics expert Zach Gemignani, but the discussion on Bolton’s blog (as well as the original LinkedIn question that prompted the post in the first place) got me thinking about this issue again. It seems to me that there are as many opinions about what makes a useful metric as there are people ready to weigh in on the subject. There’s an air of subjectivity to this whole issue of code quality metrics that is just not good.

For example, many of our customers use our products to track static analysis metrics, in other words, compliance to a set of coding standards. These standards represent a set of “best practices” that, for the most part, have been derived through (sometimes bitter) experience. But where is the evidence, some people ask, that if I write my code to these standards, I will see (say) lower defect rates?

That’s one of the questions that, over the coming months, I hope to be able to answer. Starting next month, we’ll be able to talk more about some of the fascinating stuff that’s been cooking in the labs here at Enerjy… but for now, we have more work to do!

After my post last week about the legal implications of software quality, I noticed a small piece written by David Worthington in the previous week’s edition of SD Times, detailing the British House of Lords’ proposal to increase the liability of software vendors for personal security issues. Although a recommendation from a House of Lords committee is a far cry from actual legislation, it’s worth paying attention to some of the observations in the report.

The most significant recommendation, to my mind, is that the British Government explore the introduction of vendor liability in the software industry. ZDNet quotes from the report:

In the short term we recommend that such liability should be imposed on vendors (that is, software and hardware manufacturers), notwithstanding end-user licensing agreements, in circumstances where negligence can be demonstrated. In the longer term, as the industry matures, a comprehensive framework of vendor liability and consumer protection should be introduced.

But it’s Worthington at SD Times who has the most interesting spin on this story: how do you handle liability issues in open source software? He quotes the Lords committee’s technical expert, Richard Clayton, who seems pretty sure that open source contributors could find themselves being held liable for any negligence, and even goes so far as to suggest that difficulties over assigning liability could produce a bias against free software.

Clayton ends up predicting that the end result, after case law settles down and penal code is established, will be that the software industry will become “just the same as any other industry”. Imagine that!

The Glitch Watch in-tray is bulging with reports of just two stories this week. First, from about 70 million miles away, comes the news that the Mars Odyssey Orbiter has entered “safe mode”. (I assume that’s not the same safe mode that my Windows machine used to offer me when it couldn’t quite figure out what was going wrong.) The Odyssey Orbiter’s role is to relay data between earth and the Mars rovers. Mission manager Bob Mase expected the reboot to be complete within a few days. Faster than my old Windows machine, then.

The second story is a lot closer to home, although the terrain looks kind of similar to me. It’s about delays to Homeland Security Border Patrol’s “virtual fence” system on the Arizona/Mexico border, that was supposed to have gone live three months ago. Software problems have caused delays, but Homeland Security Secretary Michael Chertoff said that they will be ready to “kick the tires again” in about a month. Sounds like they are planning a rigorous testing program.

A couple of years or so ago, I had the pleasure of meeting Peter Coffee, who now works for Salesforce.com, but back then was Technology Editor at eWeek. We showed Peter our CQ2 product, and one of his comments went along the lines of “It’s only a matter of time before we start to see source code quality brought up in litigation cases.”

Rick Mueller is a DUI criminal defense lawyer who, on his blog, reproduced the transcript of a paper given by William C. Head and Thomas E. Workman Jr. at the International Council on Alcohol, Drugs and Traffic Safety in Seattle on August 30th. The paper is titled “An Analysis of ‘Source Code’ Litigation in the United States.” The transcript is long, but makes for fascinating reading, in my opinion, because it’s the first time I’ve come across a real example of what Peter Coffee was talking about.

I found their analysis, from a purely legal standpoint, of the state of software quality to be quite interesting.

Head and Workman start out by stating something that we all know to be true, but bears stating nonetheless: “All non-trivial software has defects.” That in itself is good news for an attorney, I guess. They go on to state that, unlike hardware components, which may simply have worn out, defective software works exactly the same way in every machine that contains the same software. So, you can’t compare results from two identical machines to find out which one has the defective “part”.

Head and Workman provide a kind of attorney’s guide to software development, in which they make some interesting observations:

Specialization and multiple handoffs
“The programmer who writes the source code is usually not the scientist upon whose science the machine is based,” they say, and then go on to point out that many others are involved in the production of the finished product. “With each handoff, opportunity for misunderstanding and mistakes in the final product, the source code, increases thereby degrading the quality of the product.”

Comments
Good programmers write comments, they say, but comments “often pose suspicions about problems that have been elusive and have not been specified and may not have been corrected. They may also contradict the instructions to the computer, as represented in the source code, meaning that either the comments are wrong, or the software is not correctly written.” You can’t argue with that!

Debugging code
If anyone’s ever left any debugging code in production software, look out: “Programmers often record information that will be helpful in resolving ongoing errors and defects. An examination of the source code will usually reveal extra steps that are not necessary to the computation of the results, but which will record information that relates to actual or potential error conditions. The mere existence of this kind of activity suggests that the programmers are trying to collect additional information in order to resolve problems they have seen, but have been unable to isolate and fix.”

Testing
This section includes an interesting analysis of why testing every path through even a small program would be impossible. Their example assumes a 1,000 line program with a 2-way control statement every 20th instruction. That gives a total of 1,125,899,906,842,620 unique paths through the code, which, even at one minute to run and check the results for each path, would take one person a total of 2 billion years to complete the testing.

Defect rates
The good news is, they tell us, that we can estimate the number of defects in any piece of code by applying the average of 25 defects per 1,000 lines of code. Thus: “The estimated number of defects is calculated by multiplying the number of lines of code by 25, and dividing that product by 1,000.”

Code reviews
“A computer scientist,” Head and Workman tell us, “preferably with a background in verifying software quality, can frequently find defects in software by “reading” the source code. In addition, source code may be reviewed automatically through automated review of the source code.” “When a DUI-DWI defense attorney requests the source code for a Breath Testing Machine, it is with an aim to conduct a “code review” to look for defects.”

There’s nothing in here that we don’t already know about. But, I find it intriguing that the legal profession is starting to familiarize itself with these concepts with a view to using the weaknesses in our quality processes to question the integrity of information and analysis provided by software.

Some big names in the Glitch Watch in-tray this week. Most of the contents of the tray were given over to the Blackberry outage, which, thankfully, did not affect me as I am in England this week. But it’s interesting to me how Blackberry’s silence after their last significant outage in April is still being brought up by the press - another reminder that a company’s response to these situations is so vitally important. The San Jose Mercury News said:

Service outages have been rare in the BlackBerry’s eight-year history, but the outages that have surfaced have prompted angry backlashes against the company because of its lengthy silences about what caused them and the cryptic and jargon-laden explanations that eventually emerge.

In other news, Paypal suffered software problems over the Labor Day weekend that delayed payments to sellers of subscription services. Looks like the problem was fixed pretty quickly. And Australian bank ANZ suffered an outage in its online banking system that affected up to 1 million customers across the country. The problem was caused by an auditing tool that failed when it hit a threshold. Gotta test those edge cases!

The more time I spend talking to people about how to improve the quality of their code, the more I realize that people are looking for easy wins. We’re all stretched for time, and up against it in terms of project deadlines. No-one has time to bring in a completely new way of working.

One of the easiest wins, in my opinion, is to start a concerted effort to improve the readability of your code. All the way back in 2000, Joel Spolsky observed that “It’s harder to read code than to write it”. And, when you think about it, code is read significantly more often than it is written. Here’s what Spolsky said:

“Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.
There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong.”

I would finish that statement off by saying: They are probably wrong - it’s just not easy to read. And, by the way, Microsoft’s Peter Hallam claims that programmers spend between 75 and 80% of their time just understanding existing code. Imagine if that code were twice as easy to read - that would be a huge productivity boost.

As usual, there were a number of software glitches for us to choose from this week, including warnings from the Singapore Stock Exchange not to trust the Straits Times Index, after the index was miscalculated by about 1.3%. It seems that a “computational error” was responsible for the problem, which was later claimed to be fixed.

In other news, the Los Angeles Unified School District has been grappling with all sorts of problems with their new $95M payroll system. It seems that some employees have been overpaid, others underpaid, and some not paid at all, since the new system was introduced. 3,900 people received incorrect paychecks last month, owing to the various job assignments and pay scales that have “perplexed computer programmers.”

But our favorite story this week comes from my original homeland, jolly old England. A plan to introduce solar-powered parking meters in the sunny south west of the country has been beset with problems, it seems, because the controlling software fails to put the meters into power saving mode when they’re idle. That means that meters have been failing either because of gloomy weather (in England? Surely not!) or, because the meters were positioned under trees. The council’s proposed solution? Chop down the overhanging branches!

Recently, I was out at a client site with Michael and Rich to configure and install our software code quality monitoring tool, Enerjy CQ2. Using read-only access to the client’s version control system, the tool synchronizes the code every night, runs a number of diagnostics including static analysis and unit testing, and then summarizes the results for the development team to review in the morning.

Everything was going great until we pulled down the source for one of the modules and tried to build it. The build failed with one error – a class referenced in an import statement couldn’t be found. That wasn’t a surprise; we’d encountered this type of error several times already, and we just needed to track down the library file containing the class, copy it over to the build machine and add it to the build classpath.

But it turned out that the library wasn’t so easy to find. It was part of a commercial product, and didn’t seem to exist anywhere in the version control system. Finally we tracked down the client’s build master who found the library on their build machine and transferred it over to our machine. With the classpath updated, we were on our way again. Although the client wasn’t billed for it, we estimated the total cost of the delay, based on our hourly rates and the interruption to the build master at around $1,000.

While we were waiting for the file, just out of curiosity, Michael searched the failing source file to see why it needed this obscure library. It didn’t. The library was referenced in an import statement and never actually used in the file. If this had been our code, we could have just removed the import statement altogether without affecting the behavior of the program. That wasn’t an option here, though, since quite correctly we had read-only access to the source.

I think there are two lessons here. First, even the humblest compiler warning or static analysis violation matters. Those warnings are there for a reason. When we were writing our Code Analyzer tool, we debated whether unused import statements were important enough to include as a warning. They earned their place because, although we couldn’t see quite what problem they would cause, we had seen over and over that mistakes in your code, however benign they appear, will find a way to get you in the end. Sometimes they show up in a big way, by crashing your app in its first public demo or unnecessarily breaking a build. More often, these kinds of problems just cause little irritations and costs that slow down a team and make working on a project harder than it needs to be. The total cost of the problem swamps the few seconds it would take to just fix the code, but because that cost is spread out over time and developers, it’s very hard to quantify and we continue to believe that the problems are benign.

Second, this is a great example of why you should include all of the external libraries you need to build your project in the version control system, along with a build script. As part of your continuous integration infrastructure you’re probably already storing a build script in the version control system. It’s worth checking every so often that you aren’t accidentally relying on other software installed on the build machine by checking out your code onto a clean machine and making sure it still builds correctly.

What happens when we start out on a new software project? Typically, we sit with our customer (internal or external) and we talk about three key variables. They are:

1. Feature set
2. Budget, and
3. Timescales

Our discussions about these three variables, are open, up front and on the table. We have plenty of good conversations with our customer about them. Usually, we document them, just to make sure that there’s no confusion about what we are going to deliver.

But there’s a fourth variable that we never ask them about. We never, ever, ask the customer what level of quality they desire.

Of course, the reason we don’t ask is because we don’t want to hear the answer. Because, if we ask how many defects they would be prepared to accept, they will say “None. We want zero defects, naturally.” And then, we wouldn’t know how to tell them that there is no such thing as defect-free software. It’s like hen’s teeth and horses’ toes - it’s doesn’t exist. Even if we have a service level agreement that talks about defects, it usually only specifies how quickly we have to fix any defects that are found.

As with many endeavors, software quality is a balance between cost and risk. It is possible to reduce the risk that defects will creep into the finished product, but only at additional cost. And if costs are being squeezed (they often are), then quality is usually the first thing to suffer. And by not discussing that fourth variable up front, in as much detail as the other three, we’re doing ourselves a disservice.

So, in reality what we do is we assume we know what level of quality the customer expects. And usually, we are wrong. Because the customer is thinking “zero defects.” And we are thinking something else. (Or, quite possibly, we’re not thinking about that issue at all.) So, when defects crop up, the customer is disappointed. And our reaction is: “well, what did you expect?” And then things usually head south from there.