tl;dr — Incremental requirement definition and technical design in agile sprints allows us to use Agile methods for exploring innovative solutions for our customers. Also, your product manager and architect will love you.

What I am saying is nothing new. It is sold as “Agile Transformation” for organizations by hundreds of coaches and consultants. This is just my tech-centric, mostly jargon-free version of it.

In most of the teams that I have worked with, the Agile method is used by developer teams as a tool for improving execution excellence and visibility. We break down tasks into stories that can be delivered in a 2 week sprint, retrospect at the end of the sprint to assess what went well and what did not, and try to get better over time. Full kaizen for all things code.

However, two stages of the software lifecycle are often left outside this system. One is the product manager’s job of writing the spec, and the other is the Architect’s job of defining the overall architecture of the system. These are not bound by sprint boundaries and are often expected to be fully completed before dev teams start writing code. This BRUF (Big Requirement Up Front) and BDUF (Big Design Up Front) is what I want to talk about in this article.

The product manager gets the worst part of the deal, especially for large projects. She is expected to spec out in very great detail exactly what the dev teams are to build. She is needed to do the impact analysis to convince managers that the project is worth doing, think of a broad vision of how the project is going to evolve over a multi-year horizon for the technical architect, define the customer workflows with designers, and figure out the innumerable minutiae of the frontend/backend behaviours BEFORE THE TECH TEAMS EVEN LIFT A FINGER! Never mind that the business teams often only have a rough idea of what they want. Never mind that we are not even sure whether any of this works in the real world. All of this tremendous body of work is required to be finished at the outset before engineering even thinks that the project has started (it is still “being figured out by them” till then).

The architect fares a little better. She is the first consumer of the requirements document and has to come up with a design for solving the problem. The problem is that design is expected to be done in great detail and all at once. Never mind that the spec itself is a figment of someone’s imagination at this stage. Never mind that most likely there will be a ton of changes as we go into execution. The design has to be frozen and signed off in its entirety.

Already, we have introduced two HUGE sources of inflexibility in a supposedly agile system. It gives us a false comfort that we have figured out the entirety of what we want to build (requirement) and how we want to build it (Architecture/Design). There is no more ambiguity, no further exploration of the problem or solution spaces required. All that remains is efficient execution for which we want to use agile-like techniques. Let’s start cranking the wheel.

Agile is as Agile does

But first, a short aside on what it means to be agile.

I take “Agile” at face value — make the organization literally agile by enabling it to move fast towards a known target and to quickly change direction towards a new and/or moving target. Agile can serve for great execution, but once we free ourselves from the fake comforts of BRUF and BDUF, we can employ agile for so much more.

---

---

Agile for Innovation

The problem is that we are applying agile principles only in narrow pockets instead of applying them end-to-end. Once we internalize the idea that software is meant to be changed, that it is never “done”, then we will realize that it doesn’t really make sense to try to define the entire product right in the beginning. A better model would be to employ agile as a method for learning about the problem and solving it one step, one sprint at a time. This is not to say that we shouldn’t take a big leap when we see it, just that we shouldn’t commit to a course when we have much to learn. The incremental execution methods of agile can, in this exploratory mode, be turned into innovation as we move towards a deeper understanding of our customer’s problems without over-committing ourselves to any one course of action. We can simply allow the best solution to emerge over time.

The proposal

At the beginning of the project, the product manager should be required to present/document the current state of the union and the broad strokes of the complete solution. We should know the different aspects of the proposed solutions and the functional constraints/characteristics of all of them. We should know what metrics we are trying to improve. The architect now provides the broad technical structure of the system and the architectural precepts as she sees them now. Opportunities for re-use and platformization can be identified here, alongside the absolute top-level components and their interactions.

This is the extent of up-front work that should be done. Now we enter the sprint cycles and keep iterating over the following steps.

1. The team (including the product manager) identifies one or two most critical aspects of the problem that they should tackle in this sprint. “Tackling” includes giving full features, MVPs, workarounds, bug-fixes etc.
2. The tech team (including architects) deep dive into these few problems and come with detailed technical solutions.
3. The tech team delivers the solution.
4. The product manager adds the stakeholder feedback and other improvements to the backlog.
5. Engineers add tech debt to the backlog.
6. Lather, Rinse, repeat.

See how we have given up a fixed goal and are able to address the next emerging set of priorities at every sprint boundary. We are now building software that learns from the real world even as it shapes it. There is room to understand problems a little more and to gradually come to see the solution. There is even time to backtrack and try something else.

But if I do this, then…

When I discuss this with engineering teams, they almost always complain about the lack of clarity around what is to be done and an unclear vision. Also that they will end up doing throw-away work. Also that they will amass tech debt due to frequent changes. Let’s talk about these as all are legitimate concerns.

Lack of Vision

A vision for the product is the idea that it will solve X/Y/Z problem and that we will save money/time/effort etc as we build it. It is not a crystallized set of tasks. In fact, any attempt to do so does not set a vision but actually constrains it to only those tasks. Visions is an organic, amorphous thing defined by the metrics it wants to improve and the rough means of doing so. Hence, it can only be defined in a rough way. Please spare the product manager the quest for endless details before the start of the project.

Wasted Effort

Throw away work always happens since software is always changing. The real question is how much to throw away and how often. Would you rather build a behemoth that is way off requirements and then re-do large chunks of it or would you build what is needed, refining things as you go with some small waste per cycle? I prefer the latter, personally.

I would argue that while it develops with a certain amount of wasted work, an architecture that is built incrementally is much more robust as it is built by identifying the emergent qualities of the system and not all-at-once in an ivory tower. It is much more suited to withstand change and evolve gracefully, especially since it itself evolved over time. I am assuming, of course, that the dev team gets resources and pays enough attention to the technical aspects of the codebase as it evolves. If such is not the case, then all is lost anyway, no matter what approach we take.

---

Caveat Emptor

And here I must express an often raised lament against Agile methodology — its short-term thinking. Much has been written about how agile reduces everything to sprints and no one ends up caring about the big picture. I agree with that sentiment and it behooves us to be wary as we go about our work.

As with all things in life, there is a balance — and it rests will senior folks to keep an eye out for it. We do not need to know everything upfront — but we do need to know enough that we can see the nebulous half-form of the final thing. We can iterate every 2 weeks but should have some idea of our environment and what we are iterating towards. This bi-focal mindset is the very essence of being a good “senior” in any field of work, and software development is no exception.

I would love to hear from you about your experience in using agile techniques beyond just execution excellence. Drop a note!

Read Next : More article on Agile and agility.