At Agile and Development conferences this year some sessions have been looking at how to improve Agile techniques using failed projects as examples. Discussing some of these issues and reading blogs and articles it appears that a significant percentage of Scrum projects have failed and it is not quite the silver bullet many expected it to be.In this weeks video blog I asked some of the leading Scrum experts (Dan Rawsthorne, Alan Shalloway, Sanjiv Augustine and Ken Pugh) what they thought the most common reasons for failing Scrum projects is and what advice they can give to resolve this. In their opinion, it appears management and managing the Product Owner are currently areas to improve upon.I believe that many organizations that practice Agile techniques do not strictly follow Scrum, XP, Crystal or any of the other flavors of Agile directly, but instead adopt a hybrid ‘Agile-esque’ practice using parts of these methodologies that they feel work best for their projects. This was reiterated at Bob Martin’s keynote at Agile 2008 where he also stated that ‘Agile’ will be the term used as a practice rather than the umbrella for the individual practices such as Scrum, XP etc. Will this see an increase in Project success? Only time will tell.One opposing theory is; teams will adopt whatever practices they can implement easily and quickly to comply with senior management edicts so they can claim their organizations are ‘Agile’ (whatever that means to them?!?). The reality is, underneath they will really be working in a Waterfall manner with little change to project success.Such proponents of this theory will state that only by following the practices strictly, can you achieve project success repeatedly (as demonstrated by Menlo Innovations).

Fred Brooks’s law of ‘adding manpower to a late software project makes it later‘ is one most of us have tried to prove wrong…….and failed!I was at Agile 2008 and saw an interesting session, “Breaking Brooks’s Law” from Menlo Innovations, a Michigan based Java development company. They claimed to disprove this law and demonstrated their working environment and techniques that allowed them to do so.Although the presentation was only 45 minutes, we were in the room for almost 2 hours asking questions to determine how robust their techniques were, and to gain more insight into the conditions developers work under.Menlo’s results are based on a 3 year project that the customer had a deadline to demonstrate at a show. More features were required for the show than currently in the plan. So rather than re-prioritize, Menlo decided to add more developers to attempt to complete the work. They managed to complete the Project on time with all added functionality.The environment at Menlo is quite unique. All developers are co-located in the same large room (no offices or cubes) and pair program 100% of the time - they follow strict XP practices. A scheduling team determines which projects developers work on and who they pair with on a weekly basis. So developers work with different team members and possibly different projects every week.Also, as part of the contract, the customer comes to Menlo every week to prioritize the work for the next sprint.These techniques may appear somewhat draconian (100% paring for example). I managed to catch up with the team and interview them to discuss this project further, bug rates, staff attrition rates and how Project Managers can push the message of pairing to Senior Managements/Directors (see video).I thoroughly enjoyed talking with the team from Menlo and they invite anyone passing by to stop in and take a look at how they operate. They also have an interview process which involves a large number of candidates performing a number of tasks including Pair Programming, with an appointment you can observe this too. A detailed paper about their techniques and contact details are here.

Earlier this week on Javalobby, I posted an extract from our monthly newsletter regarding our analysis of the ‘missing braces in If Statement’ rule firing and the potential bug involved.If you omit braces and use Static Analysis tools it is a problem actually finding these bugs. Why? Because you probably have the rule turned off!Having any tool tell you of violations in your code purely due to a stylistic preference will result in thousands of false positives and eventually demotivate the developer from using the tool, therefore, for the other good causes these tools promote, in this case it is probably better to switch the rule off.Hopefully Unit Tests are in place to cover these areas if static analysis is not used. The only other way to find these bugs is manual code review (laborious, time consuming and introduces human error i.e. the bug may be missed anyway) or debugger tracing.Another interesting aspect about the post was the comments. Many of those who omit braces in If Statements do so for readability purposes. Psychologically, this may mean that these developers see readability as a higher aspect of quality than possible fault-prone code. I’m not stating that readability is not important, far from it. However from a business perspective, having the code released with less faults is a higher quality perspective.

To follow from the successful launch of our free Eclipse plugin, today sees the official launch of our Service Offerings for Java Development Teams.Over the last 5 years we have been helping development teams increase the quality of their applications.From these experiences and our own experiences we have formalized a collection of offerings that can help improve code quality further.In summary our Services consist of:- An Internal Quality Code Report- Continuous Integration Services- Context Specific Standards for the free Enerjy Software plugin.More details can be found on our Services page of our website.

Chuck Allison’s interesting article “Software: Use At Your Own Risk”, (Better Software Magazine, July/August 2008, p. 9) discusses the risk inherent in software licenses compared to other warranties, reasons for the lack of quality in software and lists a sample of the code of ethics of the Association of Computer Machinery.Here are some reasons why I believe that the Government will not take effective measures to enforce more liability on software vendors in the short or long term (within the next 10 years).1. Consumers are instant gratification junkies. Consumers want things ‘now’ and are only mildly disappointed when they have small failings, so they do not officially complain so much as in other industries. We only have to look at the lines outside Apple Stores for the new iPhone to realize this. Many people were not even concerned that some phones had activation issues, as they understood that Apple would eventually fix this.2. No physical boundaries. License issues will differ in different countries and trying to enforce this will be an administrative and legal nightmare. Also, software can be downloaded from the Internet from anywhere. Even the attempted Chinese firewall was worked around within a day3. It is understood that software is never defect free. Maybe this is not the right attitude, however it is one we have ingrained into ourselves over the last 20 years as both users and developers. Microsoft’s famous Blue Screen of Death was frustrating for developers, but after a quick reboot, work continued – it was accepted (I say “accepted” as it was the most common OS used and there was no mass exodus to UNIX, or later Linux). Would the same attitude apply if your car broke down several times on your way to work?4. Software organizations do not learn. Many articles quote from references such as Brook’s ‘Mythical Man Month’. This book was written 30 years ago and many of the failings in Software Development back then still exist today. For example adding people to late running projects only makes them later.5. Perceived insignificant importance. We know the cost of failed software is in the billions, but other things in the public eye have happened to constantly keep this from being a primary concern. For example, The Gulf War, 9/11, Presidential elections, rising fuel prices and the current Fannie & Freddie crisis. Although software is involved everywhere the quality is not considered a public priority.One of the ethics Allison lists is “Contribute to society and human well-being”. Unfortunately, for Governments to pass legislation of the sort suggested by Allison, a monumental catastrophe will have to happen. We saw something along these lines with the introduction of Sarbanes-Oxley compliance following the Enron scandal. However, even if a plane were to crash due to software error, it is unlikely that even loss of life would be enough for the Government to pursue measures. In my opinion, the outcome would be that the airline and the manufacturer would have lawsuits filed against them, while the FAA and software producer would, at best, experience some bad press.

Two of our recent blogs; The Misnomer of Best Practices and The Dreyfus Model are tied together nicely in this episode of Enerjy.tv by way of a conversation I had with Dan North of Thoughtworks.

Dan regularly speaks and blogs about these topics and gives his views on where so called Best Practices should be in an organization and how they are used depending on your position on the Dreyfus Model. Dan gives examples of how developers use Best Practices when starting a new skill or position within a company and how these practices and rules diminish in importance and become a hindrance once skills levels rise towards the Expert level.

Achieving Project success over the last 30 years has been difficult! Reports on various studies released over the last 5 years state as little as 30% of complex projects succeed. Although this number is climbing, very large projects (300 – 500 man years) have been the least successful in succeeding, with one report in 2007 (Sauer, Gemino, Horner Reich) studying 412 projects finding no successful projects of a size greater than 200 man years. So why do we continue to undertake such projects and how can we succeed?

Eating The IT Elephant – Moving From Greenfield Development to Brownfield by Hopkins and Jenkins introduce ‘Brownfield’ as a philosophy for the evolution of legacy projects. They recognize that for the delivery of large complex problems to become repeatable and predictable some fundamental areas of the IT industry have to change.

The book is split into two parts; the first half looks at problems in large projects dealing with extending legacy systems, their failures and lessons learned – Communication and embracing complexity appears to be the biggest issues.

The Second half looks at how an Elephant Eater (a system which can consume and understand all relevant information about a project and its environment) can theoretically be implemented. Herein lies a paradox – the Elephant Eater seems to be such a large project in itself that it is doomed to failure.

That said, a lot of their concepts are very interesting such as Software Archeology, transforming Legacy code and configurations into different views and models. Also, using visual environmental models to pass information to various members of the project at different levels. So high level business overviews for Stakeholders and lower level technical detail for developers/architects. The difference being that these are actually linked, so business elements can be traced through lower levels right to the code if necessary. They even suggest utilizing Second Life for this.

Their experiences of how project managers tackle Change Management and embrace Induced Complexity is reflected in their conclusion that it is not technology that causes most large projects to fail but rather Inconsistency and Ambiguity. When projects are late, the wrong solution most managers turn to is to add more people in a last ditch attempt for success.

A common recurring theme is that there is a lack of experts in the IT industry to understand the external environmental issues and constraints that often derail large projects down the road. One solution they suggest is the ‘Site Visit’ where team members can actually have experience of the current system working to determine any issues that may not be apparent in any requirements received.

The book gives some great examples of Legacy evolving projects failing because Greenfield techniques were used. Most of these failed in the later stages and they conclude that although the solutions were elegant they were solutions to the wrong problem – namely integration failings. From their experiences they introduce the VITA (Views, Inventory, Transforms and Artifacts) Architecture.

There are a few things I would have to question.

The authors seem to believe the IT industry is somewhat mature. I don’t think this is the case at all, in fact I would go as far as saying it is still immature, especially in the development lifecycle, hence few to none of these very large projects succeeding.

One suggestion they refer to a few times is to ‘use your own language’ when discussing problems with team members. I believe that this actually causes more Ambiguity and more formal languages need to be present to evolve our profession.

A suggested approach to Pattern Driven Engineering (PDE) that is built in as part of Model Driven Architecture (MDA) is a focus of Brownfield Development. This may be fraught with danger as it is becoming recognized that early use of patterns lead to them being used as a complex solution to a simpler problem – Pattern Happy Development. An alternative would be to recognize these patterns in refactoring stages and then apply them to increase the quality of the code.

For a 200 page book there is a lot of interesting and somewhat controversial ideas. I would recommend this book to any project manager or architect working, or have worked on, a large system integrating other Legacy Systems.

A large part of improving the quality of software is knowledge sharing (indeed it is one of the basic tenets of Lean). In most organizations when a bug is submitted it is usually delegated to the developer who wrote the original code.

Paul Pagel’s blog discusses how his team tackles bugs in development and production.

I especially liked:

“[The Production Support Resource] seems a lot like a silo to me. Everyone should be able to do production support on any system. I should have to, because it is a perspective of the system that is important to have. In response to this, we came up with a system of triage. Each day of the week is assigned to a specific developer. If a support item comes up, it is the job of the triage developer to respond to the client/customer we are working on it.”

This is such a simple way of sharing knowledge and keeping tedium away from a designated ‘support engineer’. However, managers may originally question this for several reasons including:

1) A faster response time will occur if the original author fixes the bug. – This is short-term instant gratification. After a while the difference in time for a bug fix between developers (of similar skill level) will most likely be negligible.

2) The author is an expert in that area. – This allows others to increase their skills knowledge, benefiting the entire team. What happens if the original author is out sick or even leaves the company?

3) We’ve always had a designated Support Engineer, everyone knows who to go to. – This is a Command and Control’ management style and in most cases leads to lower team results. Managing by Facilitation and allowing the team to assign work themselves has proved more motivational to the team and resulted in higher quality artifacts.

I strongly believe that this method is more beneficial to the team, evolving individual’s skills and ultimately benefits the customer.

One evening after sessions at the Better Software Conference, Dan North and I got into a discussion regarding the phrase ‘Best Practices’ and concluded that this term was actually a misnomer.

Let’s take a non-software analogy; wearing a seat belt in a moving vehicle.

With all the studies that have been performed over the years, one may believe that wearing a seat belt is a best practice. However, for a very small minority of cases, it is not the best thing to be wearing a seat belt. EMS and Police personnel in some countries are not required to wear seat belts, because they can respond faster unhindered by a seat belt. Also some drivers, like those in the TV show Ice Truckers, do not wear seat belts because they need to be able to jump out of their rig the moment they hear ice beginning to crack. These may be minority cases and for over 99% of us it is a good practice (never mind a legal requirement) to wear a seat belt–but it is not a Best Practice.

When we talk about Best Practices we really mean ‘the current best thing to do in a particular context’. When coding, depending on the situation, we try to solve an issue using some practice we know works effectively. Wait! Haven’t we heard this before? Isn’t this what we use to describe Patterns? On reflection, patterns would be a sub-set of what we wish to achieve and still other ‘good practices’ will need to be enforced somehow, especially early in a developers career.

The terms ‘Best Practices’ and ‘Standards’ are also used interchangeably, especially when applied to code. This is wrong! Coding standards may be formed from current ‘best practices’ but other issues such as law and external environmental concerns may mean that the code standards are not necessarily the current best way to write a piece of code.

This is just one example of the terminology problems that needs to be clarified in our profession. I don’t believe a committee can enforce this, but terminology, over time, will become more widely used and accepted as our industry matures, as has occurred in other industries over a period of time.

Many have written about language ambiguity being one of the key issues which leads to misunderstanding requirements and that a language needs to evolve so business and IT understand each other.  I would go one step further and declare that at least two formalized languages are needed. One for the Development team for lower level issues so statements such as ‘this lends itself to a Strategy Pattern’ or ‘favor composition over inheritance’ are understood by all the team members and another for Business Analysts, Project Leaders and Stakeholders to ensure no ambiguity exists in the requirements.

IMO the first language is closer to being realized than the second. Maybe interest in UML will be rejuvenated with Microsoft’s recent announcement of UML support to be added to Visual Studio Team System providing another step towards this goal.

So I was coming back from Atlanta yesterday and, as I arrived a bit early at the gate, settled in to read my RSS feeds. I picked a spot that was far enough away from the gate not to be too crowded, while at the same time allowing me to keep an eye on the TV screen showing the flight status.After a while, I noticed that the screen, which had been alternating between the flight status and a series of Delta ads, wasn’t updating any more. Getting out of my seat to go and get a closer look, I discovered what the problem was. It was only after I checked a different screen that I discovered that my gate had changed…