The Illusion of Control: Agile As A Paradigm Shift

"I understand agile is a good thing but not for what we do, our projects are time/budget limited and detailed features have to be defined and contracted at the very beginning" a CTO once said.

The myth of maintaining better control and achieving better results by thoroughly planning features' tiniest details during early stages of a software product development is not uncommon - It's by instinct that people think if they apply more control to a failed process, they'd achieve better results. It's not uncommon to hear phrases like "close monitoring", "tight process", "micromanaging" as proposed solutions to a failed process or an underachieving team. Traditional planning assumes that a detailed plan and cost estimate for the work to be done can be achieved. This assumption is not true for software development {click to tweet} and this is not how successful software that actually delivers value to the customer is made.

Traditional planning assumes that a detailed plan and cost estimate for the work to be done can be achieved. This assumption is not true for software development. {click to tweet}

Identify the Objective

The main objective of an IT project or any project/activity for that matter is to gain a value or a set of values. You use Panadol to relieve your headache not because of the technology behind its manufacturing. A person might buy a luxury car because it elevates/matches his social status, gives him the convenience, gadgets and safety he needs or believes he deserves. The companies are no different, they undergo a project for some value "higher sales, efficient operations, insightful reporting, ...etc", I have not seen a company that's after "following the plan" or after "completing the contracted requirements". However, "following the plan" is a statement you don't seem to stop hearing and certainly isn't the real objective, rather the plan is a mean to achieve the objective, and you should stop and change it the moment it looks like it's leading you somewhere else. {click to tweet}

"The plan is a means to achieve the objective, and you should stop and change it the moment it looks like it's leading you somewhere else." {click to tweet}

Embrace the Change

Unless you're duplicating the same product with the same team for the same customer, change is inevitable in software development projects {click to tweet} (as well as some of the other types of projects) - so, it's your choice, either you:

1. design your technique to neglect the need for change assuming it won't happe, or
2. design it to embrace and welcome the change for the greater good of the "real objective"

"change is inevitable in software development projects" {click to tweet}

If you choose #1, you and your client most probably are bound to fail. You might have a legal team closing all the gaps and filling all the cracks, but you're probably delivering something no one is going to use, and ending up with a customer who's not going to knock on your door again. However, if you go with #2 and you have the right team, you'll be making a software that actually delivers the value, is actually used by its intended audience and a satisfied customer with a long term relationship.

One of the customers I worked with did go through the process of creating an IT solution for what he thinks he needs 4 times with 4 different software providers without satisfactory success. He finally did it using agile and with the right mindset "eyes on the real objective/value". During his previous attempts, he failed to motivate the target audience of the software solution to effectively use it, ending with less than what he would expect to achieve with the system in hand. On the other hand, taking more iterative, exploratory, "inspect and adapt" approach into what to build is what worked for him. In the process, he learned about what he really needed, and was able to actually make it happen on budget/time. As long as we keep our eyes on the target /objective /value of the software than the plan we thought would work, success is not a far stretch {click to tweet}

"As long as we keep our eyes on the target /objective /value of the software than the plan we thought would work, success is not a far stretch" {click to tweet}

It should be known by now that large IT projects are risky and challenging. The way to deal with them is to break them into smaller, more manageable and largely independent pieces as well as to expect and furthermore embrace the change - because it'll happen.

• 17% of large IT projects go so badly that they can threaten the very existence of the company (2012 McKinsey & Company in conjunction with the University of Oxford)
• On average, large IT projects run 45% over budget and 7% over time, while delivering 56% less value than predicted (2012 McKinsey & Company in conjunction with the University of Oxford)
• nearly two-thirds of projects significantly overrun their cost estimates (Lederer1992)
• 64% of the features included in products are rarely or never used (Standish2002)
• the average project exceeds its schedule by 100% (Standish 2001)

The Not-so-agile

Some companies think they're agile if they have stand-ups and do iterations. Some teams view agile as a way to skip planning, or an excuse to fluctuate performance across iterations. Furthermore, when they fail to meet the agile promise in a way or another, upper management blames agile and falls back to the traditional planning as a resort to gain more control (they think) and thus achieve better results (they rarely do). But this is not agile.

In order to start becoming agile you have to assemble complete cross-functional teams, each of which is a small team. Those teams have to build a really clear backlog, and they have to be able to produce at least one working tested increment every iteration (couple of weeks).

Agile is not about skipping the plan. It's about planning efficiently and building a plan that's both solid enough to facilitate delivering the value and dynamic enough to be changed when it's no longer fulfilling the value you want to deliver {click to share}. Agile isn't just a set of practices that, if you install, fixes all IT projects problems instantly. Rather, it's a paradigm shift. It's a cultural makeover {click to tweet} that - based on the size and number of the interacting and dependent parts of your organization - can be implemented in waves where you see the results emerge along the way. However, You should start with the minimum: form cross-functional teams, be really aggressive about figuring out how to create backlog and make sure that those teams are reliably and predictably delivering a working tested increment of the product every iteration (couple of weeks in scrum for instance) {click to share}

"form cross-functional teams, be really aggressive about figuring out how to create backlog and make sure that those teams are reliably and predictably delivering a working tested increment of the product every iteration" {click to share}

The scope/time/budget Predicament

Conversations about how agile deals with scope/time/budget comes with the underlying objective of knowing how agile deals with fixed-price, fixed-scope contracts. The problem with fixing both price and scope is that you only have the quality variable to mess with. On the other hand if we take scrum as an example, the definition of "DONE" protects the quality. In reality almost every fixed-price, fixed-scope project goes at some point into a scope change process, which in turn changes both scope and price {click to tweet}. However, we still pretend it's a fixed-price, fixed-scope project. This being said, I believe traditional fixed-price, fixed scope projects are there due to lack of trust between the customer and vendor. It mostly leads to a lose-lose situation and thus emphasizes the original lack of trust {click to share}

"In reality almost every fixed-price, fixed-scope project goes at some point into a scope change process, which in turn changes both scope and price" {click to tweet}

To look at this from a different perspective and to examine the background behind this type of contracts: if we look at the historical situation that "you only get one chance to figuring out what you need, and one chance to getting the product you need developed" it makes sense from a risk management point of view to put a price limit on it (A.K.A budget). However, in an agile environment where the customer has the power to order the product backlog, and thus continuously gets high value functionality increments earlier - the risk is remarkably different, and it makes sense to at least implement changes to the traditional type of contracts.

Given the above, agile has different types of contracts which I'll not go through into details here, but I personally have tried the "change for free and money for nothing" type of contract and it proved mutually successful. It does have a budget and time in place but it has in its DNA the ability to morph to variable-price or variable-scope. There is also the T&M (time and material), Capped T&M, and incremental delivery contracts. Alister cockburn put a summary of the different types of contracts here.

"but how would you estimate the cost of an agile project?" one asks.

Although everyone knows (but mostly won't admit), estimates are at best an educated guess {click to tweet}, and this is not an agile problem as we've already established by now "I hope". However, agile have pretty handy techniques of estimating cost that can be reliable as a starting point if you have a well formed cross-functional team that's truly agile (see above). Points estimate against historical references, and t-shirt sizes estimation poker proved relatively reliable to base a project on. However, if you think it's more reliable to use traditional estimation techniques - where you have to break every feature into technical tasks from the very start of the project - think again. Estimate is at best an educated guess.

"Although everyone knows (but mostly won't admit), estimates are at best an educated guess" {click to tweet}

That's it for now, I'd love to hear back from you on your thoughts regarding what have been presented in this article.