Understanding what's wrong with the traditional model

Since you are reading a book on mastering Scrum, I might assume you already know something about agile practices and how they evolved to address the issues of the traditional software development model, also known as the waterfall approach. Still, it's never safe to assume. So, I'm going to start this book with this section to provide a quick overview of the traditional software development model, why and how it developed, and its shortcomings.

Modern computing became a reality in the early 1940s through the 1950s with the introduction of the first general-purpose electronic computers. The introduction to modern computing is particularly relevant to our discussions on the evolution of software development practices, as the high costs, skills, and complexity of software development drove early software and Systems Development Life Cycle (SDLC) practices.

In the earliest days of computing, computers filled an entire room, were extremely expensive, often served only one purpose or business function, and took a team of highly skilled engineers to run and maintain. Each installation and the related software programming activities were highly unique, time-consuming, schedule-bound, complex, and expensive. In other words, early computing efforts had all the characteristics of project-based work – and development activities were managed accordingly.

It's only natural that executive decision-makers and paying customers want to manage their risks in such an environment, and the discipline of project management provided a model for managing unique and complex work in an uncertain development environment under approved constraints. Project constraints consist of the approved scope of work, budgets, schedules, resources, and quality. Specifically, project management implements detailed project planning, documentation, scheduling, and risk management techniques to guide and control complex development projects within the constraints approved by the executives or paying customers. In software development, such practices came to be known as the so-called waterfall model.

The discipline of project management is as old as the Egyptian pyramids, dating back to roughly 2500 BC, and probably older than that. Project management evolved to manage work associated with building large, complex, risky, and expensive things. If you are the person financing such endeavors, you have a strong desire to control the scope of work to deliver what you want, in the time you want it, at a cost you can afford, and with certain expectations about the quality of the deliverables. Hence the genesis behind managing work under specific and approved project constraints.

Project management looks at work from the perspective of protecting the customer's investments in risky, expensive, and complex projects. In effect, customers accept a layer of management overhead with the goal of helping to ensure on-time and on-budget deliveries.

In a modern yet classical context, projects have the following characteristics:

In the traditional project management paradigm, the objective is to manage work within pre-determined constraints. The constraints are the scope of work, budgets, schedules, resources allotted to the effort, deliverables, and quality. The goal of project management is to successfully deliver satisfactory products within the constraints approved by the paying customer. The philosophy behind project management is that the combination of rigorous planning, engineering, and coordinated life cycle development processes provides the best approach to managing uncertainty and risk.

The project management philosophy works well when building things that are difficult to change once construction begins. Imagine that the pharaohs decided they needed tunnels and rooms within their pyramids after construction. I suspect it would have been possible, but the effort, resources, time, and costs would have been extraordinarily higher than if they planned and designed on those requirements before starting construction. The same concept is true when building ships, high-rise buildings, energy and telecom utilities, roads, and other large, complex, and expensive physical products. The cost of change is extraordinarily higher after construction starts.

Early computer scientists and IT managers faced many of the same constraints and issues those other industries faced. In other words, the characteristics of managing software development projects mirrored those faced in other industries that build large, complex, unique, and expensive things. It, therefore, seemed logical that software development projects also required rigorous planning, architectural design, and engineering, and coordinated life cycle development processes.

All software products and IT-based systems have a life cycle. The traditional model conceptually breaks up a product's life cycle into a series of phases, with each phase encompassing a type of related work. Phase descriptions expose work through a series of decompositions, consisting of the following:

Product life cycle processes follow a common pattern of work activities. Products are conceived; requirements or needs are defined; architectures and designs are established; the products are constructed and tested. Then, they are delivered, supported, maintained, and perhaps enhanced until, one day, the decision is made to retire the products.

In the traditional plan-driven and linear-sequential software development model, the life cycle processes are laid out to show work and information flowing from one phase of work to another, as shown in the following figure:

Figure 1.1 – Traditional plan-driven, linear-sequential "waterfall" software development model

The figure implies this approach to development is a straightforward process from beginning to end. Still, even in the traditional model, this is not entirely true. There tends to be some overlap of work among the phases. The traditional waterfall model is more of a conceptual framework than a mandated work structure.

But the biggest point I want to make here is that the more a team frontend-loads requirements, the more protracted the overall work becomes. The added work in process (WIP) adds complexity, extends the development life cycle, and limits customer input. Moreover, adding WIP delays testing that would otherwise help find and determine the cause, and quickly resolve problems at the point they arise. In short, it is that frontend-loading of work that creates all the problems in the traditional model.

There is strong evidence to support that adding size to a project under the traditional waterfall-based development model is highly correlated with project failures, while agile-based projects outperformed traditionally managed projects at every scale. For example, the Standish Group, in their 2015 Chaos Report (available at https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf), which summarized data across 50,000 projects, shows agile-based projects outpaced waterfall-based projects at every category of project size – spanning small, medium, large, and combined – with the largest projects having the greatest percentage of reported failures, at 42%.

Clearly, there are other factors at work besides the frontend-loading of requirements leading to higher failures in larger projects. Let's take a moment to see how these other factors only make things worse.

It's nearly impossible to capture all the requirements for a new product concept at the start of a project. First off, customers and end-users don't yet know what they don't know. It's not unusual for a customer to get their hands on a product and quickly discover new insights into how the tool can help them in ways not previously discerned. Second, customers and end-users have other jobs and work to do, and it's difficult to get them to sit through a requirement gathering session for protracted periods. And, even if they make the time, they may initially forget about things that later turn out to be very important.

Under the traditional model, astute development teams know they can start assessing architecture and design requirements in parallel with requirement gathering. But if we accept the premise that the requirements are not likely to be completely or correctly defined all at once at the start of the project, then we also have to believe the product designers will make wrong assumptions, leading to future reworks of the architecture, designs, and code to fix errors.

Under the traditional model, the developers follow the plan. If a customer comes in with a new or revised requirement, that change request impacts the baseline plan. The scope of work can change, which in turn can change the planned schedule, resource allocations, and budgets. If the customer insists on the change but won't change the constraints of the project, the project team will likely fail to meet their obligations defined in their project's charter or contract agreements. In other words, under the traditional model, when a client proposes a new change request, something has to give in one or more of the previously defined scopes of work, budgets, schedules, requirements, and quality.

The term scope creep refers to an unexpected and unauthorized change to the project's planned work. Scope creep is a major cause of project failures in the traditional model, because scope creep may impact the project's authorized resources, schedule plan, and the type and amount of work approved under the project's charter or contract agreements. Changes to both functional and non-functional requirements can generate scope creep. Functional requirements are the business and user requirements, whereas the non-functional requirements deal with the quality, security, and performance of the product.

To some degree, software development and testing have always gone hand in hand. But under the traditional development model, only unit and integration testing occur in parallel with coding. System testing, end-to-end testing, acceptance testing, and performance testing tend to be put off until the completion of coding for the entire product release. By that time, the complexity and size of the code base make it much more difficult to assess the root cause of any problems and fix any bugs that crop up in the tests.

The delivery/deployment processes, within the traditional model, span most of the other life cycle development phases. The deliverables of deployment processes are not just the application code but also training aids, wikis, help functionality, sysadmin documents, support documentation, user guides, and other artifacts necessary to support and maintain the product over its operational life span. Also, the development team must coordinate with the operations group to work through hardware, network, software, backup, storage, and security provisioning needs and related procurement acquisitions. Waiting to address these issues until the later stages of the project will almost assuredly delay the scheduled release date for the product.

Collectively, these issues lead to overly complex projects, lengthy development times, cost and schedule overruns, potential scope creep, poor quality, misalignment with customer priorities, and intractable bugs. Moreover, the "soft" nature of software development made it challenging for customers to wrap their heads around the cause and effects that kept leading to so many project failures. It wasn't like they could watch a building go up and see problems as they arose. Instead, a customer's or end user's first real view into the capabilities of the software is delayed until user acceptance testing, which occurs just before the software is supposed to be delivered.

As you might imagine, the folks who most take the heat for failed deliveries under the traditional software development model are the project managers and software development engineers. An astute project manager works with the development team to identify the cause and effects of their project's failures. After all, it's the engineers who are in the best position to understand the impact of unexpected changes and other impediments that prevented delivery within the approved constraints of the project.

Starting in the 1990s, a number of software engineers came to understand that the overly prescriptive practices of the traditional plan-driven and linear-sequential development model were not well suited to software development. The traditional model implements a deterministic view that everything can be known and therefore preplanned, scheduled, and controlled over the life of a project. The reality is that there are too many random or unknowable events that make it impossible to predict client needs, market influences, priorities, risks, and the scope of work required to deliver a viable product. The real world of software development is not deterministic; it is highly stochastic.

As a result, these engineers began to experiment with lighter-weight software development methodologies that provided more flexibility to address changing requirements and information on a near real-time basis. They also developed techniques to eliminate the inefficiencies and quality issues associated with a protracted software development life cycle. In the next section, we will learn about some of the better known and successful lightweight software development methodologies.