If Agile is Better, Why is Adoption So Low?

Agile software projects have significantly better outcomes than Waterfall methodologies. Statistics from the 2015 Standish Group’s Chaos Report show that, on average, Agile projects are three times more likely to be successful than Waterfall projects. This gap is even more pronounced for medium and large-sized projects. (See: What We Really Know About Successful Projects).

Even though Agile produces more successful outcomes, waterfall is still the predominant methodology. Based on data from the PMI’s 2016 Pulse of the Profession™ report, organizations are twice as likely to use Waterfall than Agile.

If we know that Agile projects are better, then why is adoption so low? There are factors that contribute to the organizations being reluctant to transform. Based on my experience, here are some of the impediments and thoughts of overcoming resistance:

There is Not a Clear, Single Path to Agile

Agile is a set of values and principles. It is not a single methodology. According to the Version One State of Agile Report, 38% of respondents report inconsistent approaches to Agile as the cause of failure.

Scrum is the predominate form of Agile with 58% adoption. However, many organizations (25%) have adopted hybrid forms such as Scrum/XP or Scrumban.   Various niche methodologies account for the remaining 17% of users.

Agile does not offer cookie-cutter solution. The lack of a clear, single path is a challenge for some organizations. They struggle selecting an approach and dedicating themselves to mastery. I have seen organizations that partially adopt Agile and/or hop from one approach to the next.

It is Easy to Cheat

Adopting Agile requires discipline and dedication. It is like to a fitness plan. Just like fitness a plan it is easy to avoid the hard work. It is easier to do a long, slow run than speed work. Similarly, it is easy to have a Scrum board, but hard to hold the team accountable.

Organizations often adopt the easy practices, but shy away from the transformative ones.   For example, over 80% of Version One respondents conduct daily stand-ups and have a prioritized backlog. But, only half have a dedicated product owner and one-third-practice test driven development (TDD).

Clients have told me they are, “being Agile, in their adoption of Agile.” In other words, they want to say they are Agile, but don’t want a disciplined practice. They are choosing the easy path, and are unlikely to achieve their objectives.

Other clients “have been Agile for years.” But, the environment retains many traditional characteristics. Daily stand-ups are an hour long. Managers dictate tasks and velocity. They practice incremental development, but only deploy to production once a year.

The Agile methodologies (e.g., Scrum, XP, etc.) do offer well-defined and rigorous practices that have demonstrated successful outcomes.  To realize the benefits of Agile, organizations need to be disciplined in their adoption.

Cultural Barriers to Agile

Culture is the leading barrier to Agile adoption and is a leading cause of failed Agile projects. According to the Version One Report:

• 55% cite the inability to change the culture as a barrier to further adoption; and
• 46% cite clashes with the existing culture as the cause of project failure.

Agile threatens traditional organizational cultures.   To be successful, organizations need to acknowledge and directly address these barriers.

Agile Threatens Management Control

Empowerment and self-management threatens management control. Most organizations still have traditional, hierarchical structures. Managers, directors, and officers derive power based on the organization they control. Ineffective management collaboration is cited 34% of the time as a cause to project failure.

Many organizational leaders (at all levels) are micromanagers. According to a 2014 Acountemps survey, 59% of employees reported working for a micromanager at one point in their career. Micro-management is at odds with self-management and empowerment.

Even in many organizations that have nominally transitioned to Agile, management control is still pervasive. I have observed:

• Line managers that continue to assign tasks to the Scrum team;
• Management oversight and control that requires a 50-page deck for a weekly status meeting;
• Extensive, manual approvals and control processes.

Agile Challenges Organizational Silos

Even though all organizations want to teardown organizational silos and barriers, they persist. Agile pushes against these barriers and requires real, cross-organizational collaboration. Agile challenges long-held notions of resource specialization, segregation of duties, and control.

A dedicated, engaged product owner is one of the pillars of Agile.   Staffing this role is a challenge for half of the organizations. Resource availability is often cited as the excuse. But, I wonder whether the reluctance of the business to commit and become a full partner in the outcome is the true reason.

Cross-functional, empowered teams challenge cultural behaviors. When delivery organizations are in silos, it is easy to play the blame game. Developers blame architects, testers blame developers, etc. Well-performing Agile teams are jointly accountable which challenges old behavior patterns.

Agile Requires Real Communication

The fifth principle of the Agile Manifesto states, “The most effective method of conveying information to and within a development team is face-to-face communication.” However, communication problems (34%) and ineffective collaboration (25%) are cited as causes of failed Agile projects.

The value of face-to-face communications has been validated by research. Work by MIT’s Human Dynamics Laboratory demonstrates this phenomenon. Changing work patterns to increase casual and social interactions significantly improves team performance.

Technology professionals tend to have personality and behavior traits that prefer indirect communication (e.g. emails, texts, chats, etc.) They tend to be introverts.   They also tend to prefer detailed documents to conversations.

I have observed teams resist co-locating or meeting face-to-face. Or, teams that gather for the daily Scrum and then return to hide in their cubes. I have heard people say, “oh, we will just assume…” rather than walk down the hall to ask.

Distributed teams pose a unique challenge to Agile. More than 82% of the Version One respondents had at least one distributed team, which is up 35% from the prior year. While technology has made it easier to bridge the distance it is not the same working together.

Collaboration tools (e.g., Wikki, chat, etc.) are poor replacements for direct interaction. Our words only account for 7% of a communication stream—non-verbal cues, tone, and gestures represent the remaining 93%. A recent HBR article found that face-to-face requests were 34 times more successful than email.

What Do We Do?

First, we should acknowledge that Agile is still in its infancy. The growth of Agile has been profound but only 17% of Version One respondents identify themselves as mature, while 33%% report being in the early adoption stage.

Second, we must recognize that Agile represents a significant organizational change that requires on-going and consistent support from leadership. It also requires the organization to commit to an on-going, multi-year journey.

Agile holds great promise and the opportunity to dramatically change how we build software and collaborate as teams. However, it necessitates the unlearning of many old organizational and management behaviors as well as software development patterns. Organizations that can embrace and adopt Agile values and principles will reap the rewards.

© 2017, Alan Zucker; Project Management Essentials, LLC

Sources:

Bohns, V. K. (2017, April 17). A Face-to-Face Request Is 34 Times More Successful than an Email. Retrieved May 13, 2017, from https://hbr.org/2017/04/a-face-to-face-request-is-34-times-more-successful-than-an-email

Denning, S. (2015, January 28). Why Do Managers Hate Agile? Retrieved May 13, 2017, from http://www.forbes.com/sites/stevedenning/2015/01/26/why-do-managers-hate-agile/

Pentland, A. & Anita Woolley and Thomas W. Malone. (2015, July 15). The New Science of Building Great Teams. Retrieved May 13, 2017, from https://hbr.org/2012/04/the-new-science-of-building-great-teams

Version One, 11th Annual State of Agile™ Report | What’s the State of Agile Today? (2017, April 6). Retrieved May 13, 2017, from http://stateofagile.versionone.com/

Welch, A. (2014, July 1). Survey: More Than Half of Employees Have Worked for a Micromanager. Retrieved May 13, 2017, from http://accountemps.rhi.mediaroom.com/2014-07-01-Survey-More-Than-Half-of-Employees-Have-Worked-for-a-Micromanager

Zucker, A. (2017, April 29). Making Sense of Agile. Retrieved May 13, 2017, from https://pmessentials.us/making-sense-agile/

Image courtesy of: http://online.husson.edu/agile-vs-waterfall/