Achieving Project success over the last 30 years has been difficult! Reports on various studies released over the last 5 years state as little as 30% of complex projects succeed. Although this number is climbing, very large projects (300 – 500 man years) have been the least successful in succeeding, with one report in 2007 (Sauer, Gemino, Horner Reich) studying 412 projects finding no successful projects of a size greater than 200 man years. So why do we continue to undertake such projects and how can we succeed?

Eating The IT Elephant – Moving From Greenfield Development to Brownfield by Hopkins and Jenkins introduce ‘Brownfield’ as a philosophy for the evolution of legacy projects. They recognize that for the delivery of large complex problems to become repeatable and predictable some fundamental areas of the IT industry have to change.

The book is split into two parts; the first half looks at problems in large projects dealing with extending legacy systems, their failures and lessons learned – Communication and embracing complexity appears to be the biggest issues.

The Second half looks at how an Elephant Eater (a system which can consume and understand all relevant information about a project and its environment) can theoretically be implemented. Herein lies a paradox – the Elephant Eater seems to be such a large project in itself that it is doomed to failure.

That said, a lot of their concepts are very interesting such as Software Archeology, transforming Legacy code and configurations into different views and models. Also, using visual environmental models to pass information to various members of the project at different levels. So high level business overviews for Stakeholders and lower level technical detail for developers/architects. The difference being that these are actually linked, so business elements can be traced through lower levels right to the code if necessary. They even suggest utilizing Second Life for this.

Their experiences of how project managers tackle Change Management and embrace Induced Complexity is reflected in their conclusion that it is not technology that causes most large projects to fail but rather Inconsistency and Ambiguity. When projects are late, the wrong solution most managers turn to is to add more people in a last ditch attempt for success.

A common recurring theme is that there is a lack of experts in the IT industry to understand the external environmental issues and constraints that often derail large projects down the road. One solution they suggest is the ‘Site Visit’ where team members can actually have experience of the current system working to determine any issues that may not be apparent in any requirements received.

The book gives some great examples of Legacy evolving projects failing because Greenfield techniques were used. Most of these failed in the later stages and they conclude that although the solutions were elegant they were solutions to the wrong problem – namely integration failings. From their experiences they introduce the VITA (Views, Inventory, Transforms and Artifacts) Architecture.

There are a few things I would have to question.

The authors seem to believe the IT industry is somewhat mature. I don’t think this is the case at all, in fact I would go as far as saying it is still immature, especially in the development lifecycle, hence few to none of these very large projects succeeding.

One suggestion they refer to a few times is to ‘use your own language’ when discussing problems with team members. I believe that this actually causes more Ambiguity and more formal languages need to be present to evolve our profession.

A suggested approach to Pattern Driven Engineering (PDE) that is built in as part of Model Driven Architecture (MDA) is a focus of Brownfield Development. This may be fraught with danger as it is becoming recognized that early use of patterns lead to them being used as a complex solution to a simpler problem – Pattern Happy Development. An alternative would be to recognize these patterns in refactoring stages and then apply them to increase the quality of the code.

For a 200 page book there is a lot of interesting and somewhat controversial ideas. I would recommend this book to any project manager or architect working, or have worked on, a large system integrating other Legacy Systems.

The first page of the preface of this book made me wince! Not because the book is bad, far from it! The immediacy of Scott’s insight into the pain of software development can only come from someone who has been there and experienced the trials and tribulations of project failure (more than once).I was expecting this to be yet another book on Design Patterns, but it really isn’t. This book attempts to look deeper into questions that cannot be easily answered and suggests a road map to evolve the profession of software development. It concentrates on practices, principles and disciplines that developers should follow when creating software, especially when thinking about how to implement features. It covers a wide range of practices, including analysis, refactoring, testing, and looks at how existing patterns should influence our design decisions.The appendix includes some very good examples of common design patterns. Different styles are applied to each pattern to teach or remind us what type of problem each pattern is used for. UML diagrams, procedural code alternatives, non-software analogies and basic OO code for implementation are included for each pattern.Since so many of us have to deal with legacy code bases, it’s always helpful when a book like this addresses that issue. Scott mentions hearing comments such as “this code is too hard to unit test,” “unit testing takes too much time” and “too many permutations to unit test.” He explains how these all point to design issues, and that leads into a great chapter discussing refactoring.Why should we refactor if the behavior does not change? This and other similar questions are covered too, explaining the concept of technical debt and the frequency of developer burnout: “Decaying, hard to maintain software will disable a development team faster than anything I know.”I would thoroughly recommend this book to any developer, however experienced or inexperienced, who wants to understand more about design patterns and how thinking in a design-driven manner can evolve our profession.I caught up with Scott at SD West, to ask him a few questions about his book.

Tom and Mary Poppendieck are two leading evangelists of Lean software development who share their experiences at conferences on a regular basis. Their first book was an excellent introduction to lean principles and tools, and this book continues the theme by explaining the concepts in the real world, and how to implement them.Firstly, let me say that, in my opinion at least, this is a book for managers who want to find out more about lean processes, rather than being aimed at software engineers or architects. And in this context, it does very well. The theories and concepts described are well laid out in individual chapters that cover: Principles, Value, Waste, People, Knowledge, Quality and Partners. The book quickly gets you thinking about how to start implementing these ideas in your own organization.For a book that is less than 250 pages, I was surprised by how long it took me to read. This was mainly due to the large number of very good and equally intriguing references and examples that compelled me to investigate further.My favorite chapter is “Waste,” including the description of the “Seven Deadly Wastes.” Sound advice is concisely provided in the form of often-missed principles that lead to waste, for example:If there isn’t a clear and present economic need for the feature, it should not be developed!Other, less obvious examples are also given such as:Rotating/replacing staff – handoffs are a good example of losing tacit knowledge (waste)Map a Value Stream - although not necessarily accurate, does identify waste from the customers’ perspective.Waste is one of those concepts that is somewhat subjective, and in one part of the book there is an example of an organization that ran multiple projects to develop different versions of the same system against a hard deadline (absolute minimal functionality but guaranteed, more functionality but not guaranteed, ideal system but very unlikely to be completed by the deadline).This can appear contradictory to a key agile principle: to develop only what you need (and indeed to a key Lean tenet, which is to reduce costs). The authors justify this by stating that one system will be implemented by the defined deadline and then something much better will be built in time for the next iteration. However in my experience, organizations tend not to have the resource available to perform simultaneous development of the same system. It is more usual to deliver to the customer as much functionality as is available by the deadline (remember that it will be working through smaller iterations) and then to implement any improvements over time depending on priority against other parts of the system.The chapter on quality begins with a discussion of the Polaris Submarine project and moves onto a description of iterative development. Then follows details of what I would refer to as implementing quality gates, code reviews, test driven development, standards and continuous integration. More detail and an example of the benefits of CI would greatly enhance this chapter.I would like to have seen more examples from the software industry to explain the many well laid out concepts. But some of the brief examples used from other industries hit the mark well, for example: Deadlines! Daily Newspapers and Concerts/Theatrical Productions hit deadlines constantly and consistently unlike the software industry.I believe the best example in the book is the Rally Software Case Study, where they explain that it took 14 months to eliminate technical debt using a Strangler approach. Some of the concepts in the book read as if these are quick fixes to implement that will see a fast return. This example highlights the fact that process management change does take time to take effect.Overall, I consider this to be a great book for laying out the principles and how to begin to implement them, especially for managers wanting to find out more about lean software development. It covers the concepts well and provides excellent further reading resources.The Poppendiecks’ companion site provides an interview about the book. Other useful resources for anyone who wants to learn more about Lean are the Yahoo groups leanprogramming. and leandevelopment.

Last week I was at SD West in Santa Clara, and on Tuesday night I had the privilege to attend Stelligent’s round table event. Around 40 people were there including industry celebrities such as Alistair Cockburn, Neal Ford, Scott Ambler and Jeffrey Fredrick.An hour was spent on the subject: “What is Agile?”; an agreed definition for which eluded everyone (even with this august group in the room).Neal Ford asked people to define agile in two words: “have fun”, “speedy delivery” and “driving quality” were popular answers. Someone commented: “selling the quality helps get agile projects underway.” Andy Glover quickly responded, “quality doesn’t sell! Speed does!”This is an interesting point. At first, the idea of focusing on speed may seem to violate many agile principles. But the speed comes from injecting quality into the development process. At first, teams new to agile techniques will be slower at producing artifacts for the customer because agile often requires a change in culture, which takes time to adapt to. However, the quality built-in will ensure that over time, releases will be quicker as the foundation of the software is a much more robust.Many times we have seen products released too soon, because time to market has been considered the most important factor, but over time code has been ‘layered’ onto this brittle foundation, causing cost to be incurred later, necessitating refactoring (or even completely re-writing) these products.The two words I would use to describe agile are “better communication”. This is something I believe is a constant in any team using agile practices for any length of time. Whether it is pair-programming in XP or stand up meetings in Scrum, communication between developers has improved with the result that (a) most team members are aware of what issues are cropping up in other parts of the project; (b) they have a better understanding of the whole system and the business benefits of what they are producing; and (c) More synergy is created and morale within the team is almost always improved.By the way, the 2008 Jolt Awards were presented at the conference, and our congratulations go to Stelligent as Continuous Integration: Improving Software Quality and Reducing Risk by Paul Duvall (CTO), with Andy Glover (President) and Steve Matyas book won the Best Technical Book category.

Lean programming has been a popular topic in conferences over the last couple of years, largely thanks to the experience and work of Mary and Tom Poppendieck. Lean programming has its roots in lean manufacturing, a management system focused on reducing waste and empowering workers to improve processes themselves. Lean manufacturing is largely based on the work of W. Edwards Deming, the statistician who revolutionized the culture and operations of many businesses by focusing on driving quality through the whole of the organization. Deming’s work was adopted and improved the results of many companies, most notably Ford, Toyota and Bell Labs (AT&T).I just finished reading Dr. Deming – The American Who Taught the Japanese About Quality, which was written by Rafael Aguayo who studied under Deming in the 80’s. The book is not, as I initially thought, a biography of Deming (although there is a short appendix on Deming’s life). It is a well-written explanation of Deming’s 14-point management system, littered with numerous examples covering a multitude of organizations and industries.Interesting points made in the book include:- Without having profound knowledge, making corrections via a feedback system is just tampering and can lead to disastrous results.- Who is responsible for quality? 90% of the things we define as ‘Quality’ are out of the “workers’” hands: training budgets, deadlines, design acceptance, tools budget and selection. These are all management issues, yet the “worker” is the one often blamed for poor quality - does that sound familiar?- Cooperation with your competitor in R&D. In Japan, R&D costs are lower because groups from different organizations are brought together to work on the technology, sharing ideas. Once they have technology figured out, competition in the market place is fierce, concentrating on features, price and performance issues.This last point may seem strange to many western development managers, but we have a great example in the Eclipse IDE project. Eclipse was created by numerous groups of people for a common cause, and then companies such as IBM and Borland compete in the marketplace with Eclipse-based products such as RAD and JBuilder.Having spent some time working as a sales manager myself, one part of the book I found tough to buy in to was the suggestion of eliminating sales quotas/targets. Aguayo presents no alternative to replace these metrics, and there may be a good reason for that - there isn’t one.Although written almost 20 years ago, this book is suitable for anyone wishing to learn more about how to change management techniques to focus on quality throughout the business. Although it is not software industry-specific, it provides some useful background to understanding many of the concepts of lean programming. Some of Deming’s management points and Aguayo’s examples may seem contradictory or even irrelevant in many development managers’ eyes, especially ‘Stable Systems’ and ‘Removal of Inspections’. This is something I will blog about some more in the next few weeks.

The relatively recently released ‘Next Generation Java Testing – TestNG and Advanced Concepts’ by Buest and Suleiman deserves a place on any self respecting developer’s bookshelf, and I personally think this is one of the best-written technical books of the year.

Obviously, this book is about TestNG (co-created by Beust) but even if you use a different unit testing framework like JUnit, the concepts and ideas are very relevant and worthwhile. Unlike some other books that feature specific tools, I was pleasantly surprised that the authors didn’t just try to ram TestNG down the reader’s throat.

The book starts with reasons why TestNG was created and some discussion of the shortcomings of JUnit, although my interpretation of the “Next Generation” part of the title is because TestNG incorporates some aspects of Integration Testing rather than pure Unit Testing which is the focus of JUnit.

Arguments for and against certain testing techniques are prevalent throughout the book, leaving the reader to determine which is best for their own situation. A good example is the authors’ opinion on TDD. Buest and Suleiman suggest that one of the potential pitfalls of TDD is that it promotes micro-design over macro-design but at the same time promotes developers thinking about exit criteria of their code.

One of the more advanced topics covered is JEE testing, and the book shows design patterns on how to test in-container applications (including EJB3) and how to eliminate troublesome aspects of existing code that are hard to test, and which you do not need to test anyway – such as JNDI and JDBC APIs.

All examples start simple and then move on to explain and demonstrate how the idea would work in a real situation. And, as I already noted, the examples are not confined to just the TestNG tool. There are good definitions of various code coverage tools and the different way they work as well as introductions to testing using different frameworks such as Spring, Guice, Selenium and various xUnit frameworks.

The authors’ view on emergent tests is interesting since, coincidently, this was a theme common with the speakers on our ‘Unit Testing a Large Legacy Code Base’ video:

“Unit tests do not have to be written before any other test…they can be derived from functional tests…Some functional tests will be written from requirements and satisfy a core piece of functionality. Over time these tests can be broken up into smaller Unit Tests to narrow down bugs.”

The final chapter, Digressions, I believe is something that should be covered in every technical book. This contains topics that are somewhat connected to the title of the book but not close enough to warrant their own detailed chapter, and also makes a very enjoyable read.

There is a rather large disclaimer at the start of the chapter stating that it is the authors’ opinion only. I found myself either nodding my head in complete agreement with their opinions or gnashing my teeth in complete disagreement.

Their opinion on pair programming is that an expert/novice pairing is unlikely to work, as the expert will get bored and lose motivation quickly. This is surprising to me as, personally, I believe that there is an argument for pair programming to be used as a mentoring technique and a way for expert programmers to help novices.

I can see this book becoming a regular fixture on developers’ desks in the future, regardless of whether they use TestNG or not. I feel this book is more suited to developers who are already unit testing and want to increase their skill set in this area and need some help on more complicated areas to test. If you currently do not perform unit testing then ‘Pragmatic Unit Testing in Java with JUnit’ by Hunt and Thomas is my recommended place to start - or you can wait for my upcoming blog post on a comparison of JUnit, TestNG and FIT.