While the advantages of agile appear obvious, I observe that many IT departments stick stubbornly to waterfall. Despite failure after failure, IT managers blame the people—staff members, vendors, users—rather than the process.
Waterfall organizes software projects into distinct sequential stages: requirements, design, coding, integration, testing. Despite its common sense appeal, waterfall projects almost always lead to unproductive conflicts. Most users cannot visualize a system specified in a requirements document. Rather, users discover the true requirements only late in the project, usually during user acceptance testing, at which point accommodating the request means revising the design, reworking the implementation, and re-testing.
In fact, users should change their minds. We all do as we learn. As users begin using a system, they begin to see more clearly how it could improve their jobs. We should welcome this learning. But under the waterfall approach, such discovery gets labeled as scope creep, it undermines the process, causes cost overruns, and leads to a frenzy of finger pointing.
Agile methods throw out the assumption that a complex system can be understood and fully designed and planned for up front. While there are a great diversity of agile methods, they almost all tackle complexity by breaking projects into small—one week to a month—iterations, each one of which delivers working software of value to a customer. Users get to work with and provide feedback on the system sooner rather than later.
So why do so many IT organizations insist on crashing over waterfalls again and again? Here are the most frequent refrains I hear from waterfall proponents:
“We need a fixed-fee contract to avoid risk.”
Fixed-fee contracts force projects into a waterfall approach that makes change cost prohibitive. While fixed-fee contracts in theory transfer risk from client to vendor, these contracts in reality increase the risk of a final product that meets specification but fails to achieve the business objectives. A far simpler alternative would be a time and material contract that requires renewal at the start of each iteration. The customer reduces risk by agreeing to less cost at each signing and by receiving working software sooner rather than later, which both achieves a quicker return on investment and assures the project remains in line with expectations.
“What’s this going to cost?”
To make the business case for a project, it is essential to nail down costs, which the waterfall method promises for an entire project as early as the requirements stage. And while I’m all for business cases, making decisions based on bogus data makes little sense. What we need is an agile business case, one that covers a single iteration and rests on meaningful cost and revenue data.
“I’m a PM and I need a schedule to manage to.”
Wedded to the illusion of predictability, many PMs identify their discipline with large, complex schedules, full of interdependent sequences of tasks stretching out over months, schedules so easy to create in MS Project but that so rarely work out in reality. These schedules, together with requirement documents the size of a New York City phone book, end up as just more road kill of the waterfall process.
“Should we buy or build? We need a decision.”
Agile eschews big designs up front in favor of designs that evolve organically over time. But IT departments must often make certain big design decisions up front, such as whether to buy or build a system or component.
And, unfortunately, these sorts of decisions fall outside of the well-known agile frameworks such as Scrum or XP, which focus exclusively on software development. So it might be assumed that once a big up front decision is required, that agile no longer applies. Indeed, it does make sense that more requirements and more design is needed up front to avoid committing dollars to the wrong vendor.
Regardless, I’d argue that there are ways to keep a project agile even if certain decisions must be made up front.
1) Keep in mind that there is no need to collect every requirement, just those needed for the decision.
2) Take advantage of demo versions and commit a few early iterations to build out rapid prototypes on different products.
3) Consider open source. Low cost means low commitment.
And once a decision to buy or build has been made, implementation can proceed according to agile principles. Most enterprise software requires substantial configuration and customization that can benefit as much as pure software development from agile approaches.
“Let’s just get on with the next project.”
Perhaps because of the pain and strained relationships that accompany failed waterfall projects, few organizations analyze the process that led them to failure. Everybody flees the sinking ship and moves onto the next project so fast that nobody reflects on the root causes of the disaster. I would think that if enough time were spent studying these root causes, it would be determined that the problem was the process, not the people.
Thoughts?
If you know of other objections to agile, I’d like to hear them. And if you have had success in introducing agile into IT organizations, please share your experiences.
