Has everyone forgotten about Waterfall? This methodology once ruled the world of software development back when teams were stable for years, if not decades. Now, when teams are changing continuously, it seems to become obsolete and neglected. Agile is taking over, be it at small startups or specialised software engineering vendors like Itransition.
Since this concept was first articulated in 2001, it has turned into the default approach to software development. In its 13th annual State of Agile Report, CollabNet VersionOne calculated that 97 per cent of organisations use Agile on their projects.
After the rigidity of Waterfall, the Agile development model came as a blessing. First of all, Agile addresses the need for flexibility. It shifted the focus from processes to teams, from time-consuming documentation to the working product, from contract negotiation to cooperation. And of course, its drastic difference was in welcoming change and uncertainty, which allowed software developers to tweak their products at any stage of the project cycle.
These overwhelming differences between the two approaches to software development divided the IT world into two camps: those advocating the traditional plan-driven Waterfall and those hailing the quirky modern Agile. Both camps have their points, but as we’ll see further, neither methodology escaped its criticisms. So let’s take an unbiased look at how Waterfall and Agile contribute to the success of IT projects.
Why change matters
With its extreme focus on ongoing changes to the working product, Agile is an easy target. The model’s antagonists often conclude that Agile-led developers just don’t know what they’re doing and prefer to figure out everything as they go in an endless loop of trial and error.
In fact, ever-changing project requirements can spring from reality checks. First, there can be disparities between the software sponsor’s vision and the availability of technical means. Second, testing the limits of technologies is likely to require amendments to the architecture or other software components. Third, playing with the product design takes more than one review and feedback session. It’s also important to consider the fluctuating market conditions, especially when it comes to products for sale, like SaaS and mobile games.
For all these reasons, just sticking to the initial plan may not always work. There’s a long way to go from an idea to a finished product, and changes en route are inevitable.
Unlike Agile, the Waterfall software development model requires that all the project stages — requirements gathering, design, implementation, verification, maintenance — don’t overlap but strictly follow one another. In such a structure, any change to a completed solution comes at a price, with reworks affecting the entire system. To avoid these massive losses of time and money, Waterfall-led teams prefer to minimise change, sometimes at the expense of end users’ satisfaction or the overall operability.
That said, the approach to change is a major stumbling block when you try to compare the two methodologies, yet there are some other predicaments. Now it’s time to dive deeper and see how both Waterfall and Agile deal with critical project success factors.
The rise and fall of waterfall
Of the two methodologies, Waterfall is the older. It originated in the manufacturing and construction industries where the sequencing of actions is the most effective way to achieve a result. It was in the 1970s that the methodology was formalised for software development practices, and then adopted by the United States Department of Defence as the model of choice for working with IT contractors.
With its roots in such predictable and regulated domains as manufacturing and public sector, Waterfall shows its efficiency where there’s a high degree of upfront clarity about requirements and scope. In other words, where stability reigns, Waterfall works wonders.
Obviously, not all industries are affected by uncertainty — some global industries don’t depend on fluctuations in consumers’ tastes and operate on predictable markets. Examples include such industries as oil and gas, mining, and space. In projects for these industries, there is more time to understand what is needed, and the changes are more likely to be minor. The project of sending a satellite into space, for example, will have the same goals in two years as it does now.
Waterfall is also traditionally perceived as a methodology for strictly regulated industries like government and healthcare. However, even within these industries needs vary.
In healthcare, for example, the project of creating databases for medical clinics will most definitely have stable requirements because the industry doesn’t change its essence. In that case, Waterfall fits the mould. But if a medical institution were to develop a mobile app for its patients to look up services and book appointments, Agile would be the path to take for the developers. The nature of the product can therefore affect the choice of methodology.
There are other reasons, though, why Waterfall is no longer favoured like before.
Why waterfall is eclipsed now
One of the reasons why Waterfall was overshadowed by Agile is the dynamic nature of modern IT teams. In the past, they were stable for many years, which is perfect for Waterfall. Also, there were fewer software development projects and fewer programmers in the world in general. Teams were not yet dispersed: often their locations were limited to the same room. This allowed for easy face-to-face communication.
All that made the scope of work much more manageable, and Waterfall thrived in those conditions. But times have changed.
The global trend of employee mobility makes projects less predictable. International teams are the new norm, while employees move increasingly between their teams from one project to another. In addition to that, people change jobs quickly and freely; they are less tied down by social benefits and pension funds, so they move from company to company, or go freelancing.
So now projects are typically carried out by teams not strongly experienced in working together because team members change all the time. Getting into a Waterfall-led team would mean to disturb a rigid system with pre-defined roles and responsibilities, where knowledge transfer would eat up billable hours not planned upfront. So when it comes to project assessment in Waterfall, it’s getting difficult to make correct predictions in such volatile conditions.
Another critical drawback of Waterfall is the lack of a working prototype until late in the project. Waterfall makes it impossible for the project stakeholder to get a sense of the result in advance. This means there can be no amendments to the initial requirements and scope, which again sends us back to the major shortcoming of this methodology: corrections to the previous work are next to impossible. If some stage goes wrong, the mistake will be inherited by the succeeding stages until the testing phase kicks in.
All considered, the Waterfall software development model is still good for smaller predictable projects where each team member gets the requirements and expectations from the very beginning. In that case, it becomes easier to onboard new team members.
However, custom software developers have been leaning toward Agile as the ultimate approach that came to replace Waterfall in many instances. Now, almost 20 years after the inception of Agile, we can reassess its effectiveness to see if it proved its worth, and whether there are downsides that hinder its project management potential.
Agile: A doubtful alternative?
It’s the axiom that Agile welcomes change and uncertainty. Agile is more flexible than Waterfall and allows for less specific requirements while project risks are typically managed better. Therefore, longer, more complex object-oriented projects work better with Agile.
Back when Waterfall ruled, information wasn’t traveling as fast and the markets were not that volatile as today. Now, depending on the industry, weekly and monthly disruptive changes are casual, so tweaks and alterations to the development process are inevitable. As a result, the end product can turn out to be absolutely different from what was planned. In these conditions, flexibility becomes paramount, especially for longer projects, and Agile is perfect for that.
However, the way Agile addresses the disadvantages of Waterfall has its own flipside.
Cultural clashes and communication pitfalls
Resembling a journey with a potentially unknown destination where the route is defined as you go, Agile calls for a cultural shift within teams. Not every programmer is a good fit for an Agile project since this requires a very particular mindset. So, cultural clashes may happen.
That said, mutual understanding and trust between team members is a must. Communication flows are as critical as allocating time for daily and weekly meetings. In Agile, messages should be conveyed in a straightforward way, and preferably via face-to-face discussions. However, this is not always possible, especially in distributed teams.
Faster deliverables, lack of documentation
Agile projects may be fast-paced indeed, but this created a common misconception that Agile-driven cycles are generally shorter than Waterfall ones. It’s not really true — it’s just that in Agile workable chunks of code are delivered much earlier in the development cycle while you’d need to wait till the final stage in the case of Waterfall. Both Agile and Waterfall projects can last for years depending on the project complexity and dynamic. Such phases as maintenance and support can be ongoing and last as long as the software is in operation.
Yet the misconception that Agile projects can only be short-term can lead to the lack of documentation produced along the way. Further into the project, this could impede knowledge transfer, especially at the handover stage when the IT vendor’s part is complete and it’s time for the software owners to take charge.
Undefined success criteria
The fluidity of Agile is reflected in its lack of quantitative measurements of a project’s success.
Waterfall has always been squeezed in between the metrics of time, budget, and scope, but Agile is less definitive about all the three parameters. It doesn’t provide any benchmarks neither, except for how workable a product release is.
It’s also difficult to decide when an Agile project is really over but by relying on a subjective perception. Potentially, iterations can be endless, and there’s always enough room for perfection. Could it be otherwise with so many changes creeping in all the time?
Call for optimisation
This makes it clear that Agile as we know it calls for augmentation or, rather, optimisation.
Agile came about as a reaction to the drawbacks of Waterfall. But nearly 20 years after the Agile Manifesto saw the light, the ideas of the limited group of its founders are bound to fall under scrutiny as they are tested by hundreds, if not thousands, of teams worldwide. Some CIOs even went on to call Agile a passing fad that will go out of fashion as software development teams recognise the framework’s inefficiency.
Of course, there’s little ground for such loud statements. But some extra effort is required indeed. First, to set up effective communication models for distributed teams. Second, to instil a sense of responsibility for the project success in each of the team members. This way, Agile-led teams will be able to optimise their workflows and cooperation patterns — something that the Agile Manifesto originally implied.
Hybrid approach as a compromise
Nowadays, various blends of Waterfall and Agile methodologies became widely adopted among software development teams. Such hybrid approaches came up to compensate for the drawbacks of both development models. Inevitably, classic Waterfall projects are starting to incorporate more flexibility, while Agile-led teams are working to provide more clarity and predictability for project stakeholders. In both cases, we get a methodological blend that helps to reduce managerial anxiety that comes from the low predictability of project outcomes.
So, agile or waterfall?
Now when both methodologies hit their maturity, it’s clear that neither Agile nor Waterfall is perfect.
To make them work, project managers need to identify their priorities first to choose the right option. Agile is ideal for volatile environments where the classic criteria of fixed time, scope, and budget are not applicable. At the same time, it is clear that Waterfall is here to stay, serving well for projects with a high degree of predictability and independence from market changes.
Managers also need to recognise that flaws don’t usually come from their choice of a methodology but rather from how the team applies its principles in practice.
That’s why it’s reasonable to look at the team’s experience with the chosen methodology and at how the team members fit together. Whether they worked with the methodology before and have good ‘chemistry’ with each other is also important. Understanding this might help to overcome potential pitfalls resulting from methodological limitations.
Alex Paretski, Knowledge Manager, Itransition