Agility Defined
There is a lot of talk these days about Agile. Agility in delivering software is vying for the position of Holy Grail in the information technology industry. As an old (trust me on that one) ex-developer, I am not trying to knock Agile (although the older I get, the more difficult it gets to be agile — physically, I mean). Agile methods, properly applied, solve a host of problems associated with delivering the technology that the business needs and deserves. Over the span of nearly 40 years in this fascinating industry, I have seen the rise (and, some would say, fall) of several similar attempts at achieving this lofty goal. I believe that if we look at the promise and the pitfalls of those past attempts, we can apply the lessons learned to predict how Agile will succeed — and where it might increase the risk of project failure.
Waterfall Methodologies
Structured techniques for delivering information technology began in the early ‘70’s to address the problem of delivering software faster, cheaper, and better (sound familiar?). The idea of the time was to replace the chaotic approach that developers tended to adopt as a natural expression of their individual genius with a set of (more or less) clearly defined rules for writing code that others could understand (and potentially maintain). The success of the structured techniques ultimately led their incorporation into waterfall methodologies that defined the development process in a series of sequential “phases” (typically planning, analysis, design, development, delivery — or some derivation thereof; an example of a waterfall methodology was Arthur Anderson’s Method 1.)
The basis of waterfall methodologies was that each phase had to be completed and sign-off achieved before the next phase could begin. The somewhat derogatory nomenclature “waterfall methodologies” suggests that the software development process, like water, can only flow in one direction. Organizations that adapted the premise of these methodologies (that structure wins out over chaos) without dogmatically applying this artificially imposed restriction were quite successful at developing and delivering working software on a massive scale. These implementations are referred to as “structured methodologies”. Overly rigorous methods failed miserably whenever process won out over common sense and practitioners were forced to “dot all their ’i’s’ and cross all their ’t’s” while patiently awaiting approval at each step of the process.
In the end, structured methodologies (by the way, still alive and doing well in some large organizations today) work great when applied intelligently on projects that have to deliver a large, coherent working system that cannot be developed piecemeal. In particular, for mission-critical systems or for systems where the failure of any component could have devastating consequences (i.e., military systems or systems that run nuclear power plants). In those situations, I will take structure over freedom of expression every time.
Iterative Approaches
Since the “waterfall methodologies” tend to take a long time to deliver working software (which might not always be a bad idea), the approach of breaking the development process down into smaller chunks evolved. Through the ’80’s and ’90’s, the concept of iterative methodologies evolved. These methodologies accepted that you still had to go through the phases “plan, analyze, design, develop, deliver“, but you did not have to solve the whole problem before delivering something the end-user could use. The premise was that the project would approach the problem of delivering working software as a series of releases in which all the phases were executed quickly to deliver something that worked while recognizing that it was not the final solution. As soon as a release went into production, the whole process started again, now given a new reality in which the implemented functionality existed and you had to plan the features of the next release. The Unified Process (UP) is a current implementation of an iterative methodology.
These iterative methodologies help organizations deliver highly complex systems that can be “grown” in a series of releases to realize their full potential. Many of the solutions that these methodologies help to create would have failed miserably using the “waterfall methodologies” described earlier. The duration of the project from start to end would have caused the pace of change to exceed the pace of progress and the development team would never have been able to keep up. The iterative approach is certainly a viable option for large-scale, operational systems that require routine modification to effectively support a changing business environment and for applications that can be delivered incrementally. It is interesting to note that the waterfall is really a single iteration of this approach.
The Agile Universe
Agile originated with a group of software developers/methodologists (who were also, apparently, skiers) in 2001 in the mountains of Utah. The Agile Manifesto is readily available on the Internet for viewing and a significant number of publications describe its virtues and implementation in detail. I do not wish to debunk any of the espoused benefits and features of the methodology and it would far exceed the scope of this article to describe the Agile philosophy in detail. Any approach that deserves the title Agile, however, is based (at least in theory) on four guiding principles that were the product of the 2001 skiing/methodology development party. Since these principles are the essence of the methodology, I will present them unfiltered and then indicate where I perceive potential risks in the implementation thereof.
- Individuals and interactions over processes and tools.This principle addresses the fact that it takes a village — excuse me, a team — to develop software and the better the team works together, the smoother the process goes. That is a great idea regardless of which methodology you apply. Software development is a people-oriented business and people who are enabled to work together toward a common goal are certainly more likely to succeed. The major problem with teams in any methodology, however, is the individuals that form it. Personalities, egos, and agendas can undermine the efforts of any team and appropriate processes and tools sometimes can counteract that threat. Because Agile is strongly team-focused, these issues have to be solved for it to succeed. Balance is needed.
- Working software over comprehensive documentation.This, too, is a great idea up to a certain point and has been the basis of all methodologies when applied with common sense. If any methodology does not deliver working software, it was not the right approach for the project. The risk has always been and still is misuse by developers — who can be notoriously reticent to write in any language that a computer cannot interpret. This often leads to the delivery of unmaintainable code. The minimal amount of documentation required to serve both the delivery and maintenance of the software — which is mandated in Agile — have to be stringently enforced. Otherwise, maintenance in the long-term could end up costing the organization much more than it can afford —and ultimately lead to undesirable outcomes (such as a bankruptcy court).
- Customer collaboration over contract negotiation.Phenomenal idea when the situation is right; high-risk in undertakings in which legal liabilities take precedence over undocumented agreements. In the end, any project that neglects the needs of the end-users, subject matter experts, and customers will fail. How we capture, clarify, confirm and complete those needs is a question of the project, the people, and the organizational culture (or, in today’s development universe, all of the cultures of the various organizations involved) in which the project is executed. Collaboration requires a high degree of trust amongst the various stakeholders in any methodology. Without that trust, we are sentenced to fall back onto the legal system. Mores the pity, collaboration is sooo much cooler when it works.
- Responding to change over following a plan.This principle acknowledges that change is the reality of any project and it takes precedence over the original plan. Actually, Agile expects change to be inevitable and anticipates it, which for me has always been common sense. Should I discover that wolves block the route I planned to take to Grandmother’s house, it would not be wise to plunge blindly ahead, following my plan straight into the belly of the beasts. I think a change might be advisable. Any plan based on the assumption of execution without deviation is an affront to the intelligence and even common sense of those who implement it. Change is real. The cheaper, better, and faster that we can adapt our plan to the change without losing sight of our original goal, the cheaper, better, and faster we get to where we want to be. This principle has been the cornerstone of every successful project since the building of the pyramids (I am assuming that — I am actually not that old).
What Is the Point?
In the end, the proof is in the pudding and being “Agile” has certainly established itself as a viable and even preferred approach when the situation warrants it. As the ” Baby Bear” of the methodology family, we look forward to helping it grow and evolve as it is tried on more and more projects. I do not anticipate that it will eliminate the need for the other two, however. The reality of software development is that there is room for all three approaches to coexist. Any organization that mandates any one of these approaches for all projects is asking for trouble. Selecting the right methodology for the individual project is a critical decision. The people involved, the project goals, and the business environment are all factors that need to be weighed to determine which methods or methodology are most likely to succeed. In the end, the delivery of a working system that meets the needs of the current and future business community on-time and within schedule will only confirm that you made the right choice — this time around.
