Whatever happened to ...? -
comparing Agile and non-Agile approaches

There are certain things that are essential in every software project such as: understanding what is needed; building something that solves the need; ensuring the solution is fit for purpose.

If an approach failed to do these basic things, then it would not be a valid approach to developing software. So it should be obvious that these things are all part of Agile or it would not be as successful as it is. In fact, Agile was introduced to make all these things better than traditional waterfall approaches. People who complain about Agile are often complaining because they do not realise how these things are handled now. Stuff might look different, but the results are more effective than waterfall approaches. As with much of Agile, a better understanding of what it is really about can help alleviate fears and concerns.

So, let’s start with a few terms that are familiar to everyone who has ever been involved in traditional waterfall software design.

• Requirements Capture - analyse the problem, the business need, the stake-holders and users (sometimes); produce a document which describes all ways of using the desired solution and all the things it must do to satisfy everyone and be 'accepted' by the organisation
• Design and Specification - starting from the previous document, design a solution. This includes making architectural decisions, producing design documents for all components (including data models, flows of control, user interface components, interfaces with other machines etc.); writing detailed specifications (to the point where any programmer should be able to take a piece of specification and write a piece of code which satisfies it)
• Coding - take the documents from the previous section and develop a plan (see below) for what will be done and when; allocate bits of specification to teams and individuals. Those people then develop code with progress tracking as described below
• Testing - design some tests and run them on the code to make sure that it does what was expected. There is a lot more to this but you get the general idea
• Deployment - put the code out there and make sure it works to satisfy the original requirements

Note that I am not saying that any of these things are bad or should not be done. They have evolved over many years as a way of developing software. The single biggest problem is that these things are done in sequence (which is why we call it waterfall) with limited opportunity to revise and adapt. There are lots of other issues with the approach. You can find plenty of articles about the advantages and disadvantages of waterfall, so I do not need to cover them again here

Now let's add a few words from project management

• Forward Planning - includes many of the things below, but basically looking ahead and planning work to ensure the project delivers what it should, when it should. In traditional approaches this is based on the requirements and specification documents. It attempts to be a complete plan which can be used to schedule work for the team, monitor progress, report to management and enable deviations and changes from plan to be managed
• Deadlines - dates at which certain pieces of work are expected to be completed. It is important to distinguish 'absolute' deadlines (e.g. times when things must be delivered to satisfy regulatory requirements) from 'project management' deadlines (which are comparatively arbitrary but are a helpful technique for monitoring and managing progress and reporting)
• Deliverables - things which will be completed by a certain date. Normally this will mean they are then available for testing. Deliverables are commonly linked to deadlines
• Workpackages - a core technique in project management. It involves breaking work down into small(ish) lumps. Planning then involves scheduling these lumps (often with deliverables and deadlines signalling the completion of each workpackage). This facilitates planning, progress monitoring and management of dependencies and resources
• Dependency Management - it is the nature of all but the simplest projects that some things have to be done before others. Dependencies might be logical (e.g. you cannot query a database until you have written it), resource based (e.g. we can only get access to a User Interface designer between certain dates; we cannot build a solution in platform X until we have a licence for it), or external (actually, this is mostly the same sort of dependencies but the difference is that you have much less control over external suppliers)
• Resource Management - completing the right workpackage at the agreed time depends on having the right resources available. This includes accounting for staff holidays, borrowing experts for short periods and ensuring any particular physical kit is available when needed. Project managers have to think ahead in their plan to ensure that appropriate resources are available at the point when they are needed and to adjust things accordingly as (inevitably) these resources fail to materialise as agreed

All those things (or equivalents) are present in Agile. Is that a surprise? If they weren’t possible in Agile then it would not be an effective approach. These things are handled in different (and generally better) ways, though. A big part of the purpose of Agile was to improve these things. Older approaches were designed when computer time was expensive and people were cheap. Since the 1970’s this balance has changed completely. There are other factors too, but that is sufficient for this article.

Let's explore a few of the issues arising from these approaches before we look at how Agile can help us deal with them

1. Issues for traditional approaches

Perhaps the hardest thing for people to accept about traditional approaches is that those approaches were often dependent upon guesswork, wishful thinking, concealment and sometimes downright lies.

So how does that relate to Waterfall and Project Management ideas?

Waterfall

In the waterfall list, the basic problem is assuming all these steps are sequential. Even a small amount of thought shows that this does not make sense. It could take months or even years to create the documents for each stage. The likelihood that you have accurately maintained information over that time, so that you deliver what was originally wanted is vanishingly small.

The most obvious issue is ‘requirements’. Can you really capture everything that is needed for a solution in one go without building anything or showing options to stakeholders? The answer is no for anything except the simplest projects, so we had a proliferation of huge ‘specification’ documents which tried to cover all the bases and were never possible to read and implement in detail. There are times when up-front requirements can work – e.g. making an interface between program A and program B or duplicating the functionality of legacy system Z with new technology X - but these are rare occurences. Sadly, a lot of companies – especially big consultancies – still promote up-front requirements documents so they can do the work and get it signed off (so they get paid) even though it is if limited real value.

Project Management

For the Project Management side, I blame MacProject (well, not exclusively). It came out in 1984 as a graphical way to plan projects on the computer, with visualisations like Gantt, PERT, timelines and so on. Microsoft Project followed 6 years later and eventually became widely used. The problem with these tools is that they give you very nice professional looking views on a project, so it is easy to think what they say is ‘true’ rather than just a guess. You can put bad information in and still get a realistic looking (but not actually realistic) project plan. These tools create a distance between the actual work and the management – and the further the management team is from the actual work, the more they are dealing with a ‘fantasy’ work model.

Even without those tools, there is a problem. You can produce monthly reports, charts for timelines, Gannt and PERT, RAG status lists, Risk Assessment matrices and so on but they are problematic for 3 obvious reasons:

1. Bad data makes visualisations unreliable
2. Data filtered up a hierarchy loses context and becomes mis-interpreted (e.g. a ‘maybe’ date somehow becomes a definite commitment)
3. In middle management (particularly) it is possible to lose focus and think that the important thing is to deliver the right graphs and reports, not the right project

Sorry, but they are all real and happen a lot. I am not trying to destroy the idea of project management or question the value of staff in these areas. It is just that, as with other aspects of software development, there are potential problems that arise and which Agile tries to address by changing some of these processes.

2. Agile is Empirical

The examples above illustrate part of why Agile was developed. We knew these things did not work. They were not compatible with modern IT Infrastructure, were based on unrealistic models and encouraged passing off responsibility rather than owning what you do. They led to poor and uninformed decision-making, poor planning, poor progress tracking and a permanent ‘fire-fighting’ model of development.

A key driver of Agile approaches is making empirically based decisions - based on hard facts and sufficient information rather than opinions or 'gut feelings'. Many aspects of Agile follow from this one basic point.

Here are a few common things to all Agile approaches:

• Problem decomposition - One of the major issues with traditional approaches is to try and capture all requirements, designs and specifications up-front. This involves taking a big problem and trying to work out the complete solution for all situations in advance. Programmers have never really worked like that, but managers often do. This is a big change in thinking. Given a problem, the first thing you do is break it down into smaller problems. Some will be obviously solvable, some will need further research, and some cannot be researched further until you have solved some of the other problems. Good programmers naturally work that way. Agile approaches extend this to all levels of the project
• Iterative/Adaptive/Responsive – Except in simple cases you cannot design and specify a product up-front. It was mad to ever think that you could. There is some upfront design and planning, obviously (though some Agile ‘purists’ do not think that is needed). In a large organisation you would have to get some idea of what you want to do, make a case, get a budget and a high level plan before an agile team to deliver it even exists. A team should be involved as early as possible so that you are connected to reality. The tricky balance is to do ‘just enough’ up front. We know complete design/spec up-front does not work, but you need to do enough to get started. So. We do just enough up-front. Then we build our systems a bit at a time, show them to users and stake-holders and get feedback. This is normally a short cycle like 2 weeks or so. That enables us to capture requirements and change designs so we ensure that we deliver what the users/stake-holders really need. It is particularly important with new technology because these people will not realise the possible unless we show them. Basically, if you ask users of system X what they want it will pretty much be system X. Do not get confused though. If our users ask for X then we will consider that – it does not mean we will do it
• Empirical – We use real data to inform our decision-making, planning and progress tracking/reporting. We do not accept someone saying do X now instead of Y unless there is a rational and justified argument. Because we use real data, we also accept that there is uncertainty. For example, we can estimate something such as effort or delivery date (and we are clear to call it an estimate) but we know it is a best estimate based on what we know now. We will always revise and improve our estimates as work progresses. This hugely improves the accuracy of our project planning as the project progresses
• Transparent – there is nowhere to hide. We emphasise availability of any level of information to anyone at any level in the organisation. Of course, most people do not want or need to know the details but we make sure an appropriate level of information is available to all stakeholders with the option to ‘drill-down’ into detail if they wish. This helps everyone connect to the ‘real’ project rather than only seeing abstractions interpreted through other people. We try to do this with ‘live’ dashboards and tools so that we eliminate the channels of hierarchical reporting with their opportunities for mis-interpretation and mis-representation
• Shared responsibility - working as a team in important. There is no concept of 'that is not my problem' or thowing things 'over the wall' to someone else (as is ingrained into waterfall models). The team work as a team, making joint decisions, taking joint responsibility and keeping an overview of the whole picture. This enables them to be self-organising (and, to varying degrees, self-managing). Transparency, empiricism and devolution of decision-making to the appropriate level are all key to making this work

An Agile team is not a team of programmers. They do not write stuff to a specification and hand it over. They are integral in every part from initial concept to supporting the live product. They have many skills which are not ‘code monkey’ work and participate in requirements, specification, design, testing and delivery. Mostly all at the same time. An Agile team has devolved responsibility and can make many decisions for themselves. If you are not Agile yourself, then you still have to learn how to work with these teams, because it redefines what you do

Let’s revisit those old words and see how they relate to Agile.

3. Agile approaches to Waterfall issues

All these things still happen, but we recognise that is was wrong to do them sequentially, so we do just enough up front and then continuously refine and improve them as the work progresses.

• Requirements Capture - Capturing requirements and delivering solutions that meet user and stake-holder needs is central to Agile working. However,we know that this cannot be achieved as a monolithic process which is done in advance of other things. We talk to people and get initial requirements at the start of a project - this includes things that used to be add-ons like privacy, security and testing. We do not attempt to capture everything in detail though. Rather than requirements, it is easier to think about objectives and constraints upon what we do. We capture these at a high level and involve everyone relevant as soon as possible (incluing stake-holders and team members).

  Once we have the big picture we can go down into details. We do not try to get all the details in advance, but continuously engage with stake-holders and users to capture what they want and show them what is possible. This means that what we deliver will be flexible but always close to what people need.

• Design and Specification - Before a team will agree to carry out a piece of work, it must be fully defined and specified - including how we will test it. Bear in mind that by 'a piece of work' we mean something very small here. We call it 'Ready' when there is sufficient detail for a team member to pick it up and build it. Things which are not 'Ready' will not be taken up and worked on by the team. However, because we work in short cycles with feedback and opportunities to change things this does not mean that we expect evertything to be fully defined at the start of a project. We break the proejct down into very small pieces and these are the things we expect to be ready at the point where we are about to start work on them. Ideally, we will have enough work 'ready' to keep us going for a while, but not for too long. If we start writing all the details too far ahead then we are back to doing waterfall by a different name. The key is to design and specify 'just enough' and 'just in time'. We would expect to have 'big blocks' of work mapped out at the start of a project but not in detail. Filling in the details is part of what we do as we progress in our project. This means that design and specification - often written as 'stories' - are a continouous and ongoing process. There will only be a complete specification of the whole solution after the project has been completed
• Coding - In an Agile world programmers are not 'code monkeys'. A 'code monkey' takes a detailed specification and turns it into code. It is a derogatory term, and intentionally so. Programmers are more important than that. A proper programmer/developer understands the big picture of what is being delivered, contribues to requirements, design and specification, and is an active decision-maker and participant in what the team does and how. They are integral to every part of of the process with one exception - they cannot decide on business values and how they affect work. That is why we have to have Product Owners
• Testing - To do successful Agile, testing is integral - and not just once, but all the time. We deliver things frequently and everything we deliver must be tested. This means that manual testing is a non-starter. If you are still using those approaches then you will need to change them before you can be Agile. This also means that tagging testing on at the end (which also means it is the thing most likely to be cut as deadlines approach) is not viable. It is essential to define automated tests which can be run before every deployment and which ensure there are no problems introduced by new code or interactions between new and historical code
• Deployment - Agile sets very high standards for deployment. Normally,we would expect the same team involved in development to be involved in deployment. Agile is fast, but not reckless. Testing (as described above), live monitoring and continuous involvement of the development team are essential. As with other aspects of Agile, there is no concept of sign-off. You can never say "development is finished so any issues are someone else's problem"

4. Agile Approaches to Project Management

Project Management looks different but still deals with the same essential issues as it always did. Learning to think in this way is one of the hardest things for people in ‘traditional’ roles, but is also an excellent path to career enhancement.

• Forward Planning - teams work with small, well-defined and specified units (often called stories and epics). These are written, evolve and change as the project progresses so they are not useful for planning (but that is a common mis-conception). For planning purposes, we create bigger groupings above this (sometimes called features, but it does not really matter what you call them - I call them 'planning stories'). These are more stable and can be used for all the normal things that would be expected in planning. They are not fully defined, but have sufficient detail to let us do a lot of the planning, progress management and reporting activites that you would expect. In a well developed approach electronic tools dynamically map and monitor these levels. This enables us to replace report writing with live dashboards - offering a much more accurate and fact-based view on our project

  Agile folk mostly use a 'backlog' of work to be done. That is where forward planning lives in the Agile world. In particular, the backlog describes (at varying levels of detail) what needs to be done. Items from the backlog can be allocated to future sprints. The combination of setting sprint goals and allocating backlog items to future sprints is how we manage forward planning. Planning 3 sprints ahead in detail and maybe 10 sprints ahead in general terms is fine. But it is flexible and not a commitment. Planning further ahead in detail makes us less agile and more waterfall because we become less responsive. Planning only 1 sprint ahead is less Agile because the team is just being 'drip fed' work rather than being empowered to organise it and make decisions for themselves

• Deadlines - 'Absolute' deadlines are still just as important as they were. Deadlines used as a project management and progress tracking technique are no longer a valid way to do things. The team organises its work in a dynamic manner to ensure that it meets the absolute deadlines. Other deadlines are replaced with things such as sprint goals and alternative ways of tracking progress
• Deliverables - Forget these. Agile teams work on short, iterative cycles where each cycle will end with 'potentially releasable' content. The result is that 'deliverables' are produced and demonstrated on a very short cycle so the creation of arbitrary deliverables is unnecessary. Deliverables linked to absolute deadlines still exist, but the nature of a deliverable is not necessarily as fixed as it would be with traditional aproaches. The responsive nature of Agile means that what is delivered may differ from initial expectations, but will probably be closer to actual needs and solutions
• Workpackages - this is an obsolete concept. Working with short, iterative cycles mean that the work is naturally grouped into small time periods with well defined objectives and clear success/failure
• Dependency Management - This is much enhanced in Agile approaches. The team works with stories which it picks per sprint. They will not accept stories into a sprint which have unresolved dependencies, so we have a very detailed level of dependency management

  By using higher level structures (e.g. planning stories) we can also do high level planning and management of dependencies. This gives us much greater power and flexiblity than traditional approaches

• Resource Management - the use of 'Planning stories' enables us to manage external and visiting resources at the high level very effectively. Teams plan at a level of detail which enables them to handle things such as holidays and unavailability of staff/resource. One advantage of this is that teams review plans of a regular basis so they can change what they pick from their backlog to suit available resources and maintain efficiency