“Lies, damned lies, and statistics”…Mark Twain
Project managers need to be very careful how they use probabilities and statistics when discussing their projects. Individual projects do not conform to the laws of statistics, and statistical references in the context of an individual project are often misused or misconstrued.
In July 2014, National Public Radio (NPR) produced a five-part series on “Risk and Reason” where the use of statistics and probability in daily life was examined. In one episode, they asked listeners what it means when the weather forecast was a “20% chance of rain tomorrow.” Only 51% of the poll respondents correctly answered it means that it “will rain on 20% of the days like tomorrow”.
Weather forecasters are notoriously bad at predicting tomorrow’s weather. The fault does not lie with meteorology. Predicting the weather is difficult because there are so many factors that contribute to the weather events on a specific day.
In Chaos Theory, the Butterfly Effect describes the impact of a seemingly inconsequential event (the flap of a butterfly’s wings) on future significant event (a hurricane several weeks later on the other side of the world). The point is tiny changes in one variable can significantly alter the predicted outcome. Projects are like the weather many variables impact the success of a project and most are difficult, if not impossible, to measure.
Project managers can also use descriptive statistics to explain observed events. For example, according to a recent PMI Pulse of the Profession™ survey, slightly fewer than two-thirds of projects meet their goals and business intent. While most of us are aware of this or similar statistics, few of us expect that our next project has a 1 in 3 chance of failing.
There are things project managers should consider when using statistics and probabilities:
- Descriptive statistics (e.g., average duration of a project) need to be used carefully when describing the attributes of a group of projects to ensure the statistics support the circumstances being measured.
- Predictive statistics (e.g., likelihood of project success) should not be used when discussing expected outcomes for a specific project because each project is a unique event—like today’s weather.
- People do not uniformly interpret statistical data and statements, and their reaction to statistical information will be filtered through their personal risk tolerance. In other words, the project team members will react differently to the same statistical statement.
Descriptive Statistics
Descriptive statistics are helpful when trying to comprehend large volumes of data. When using descriptive statistics, it is important to be clear on the intended use of the statistics because the context of the comparison needs to be relevant.
For example: if we want describe the height of men that play basketball, it is important to consider how we want to use the statistic. The average height of all American males is 5′ 10″, but the average height of professional basketball players is 6′ 7″. If we were discussing a 6’ 4” player, it would make a big difference if he were playing in a neighborhood game or the NBA.
As project managers, we often are presented with data. When analyzing and presenting this data, it is important to understand the intended use and the context of the data. For example, if you are looking at trend data (e.g., number of defects over time), has the information been normalized (e.g., divided by number of projects or code streams)? Descriptive statistics can be very helpful to a project manager when used correctly.
Predictive Statistics
Project manages should never use predictive statistics when discussing their specific project. Predictive statistics are useful for estimating outcomes for a large number of controlled, repeatable events that occur many times. However, individual projects do not meet these criteria.
Rolling dice is an example of a controlled, repeatable event. Each time you throw the dice, there is a 1/36 or 2.7% chance of rolling snake eyes. However, that doesn’t mean that you will only roll snake eyes once in 36 attempts—you might get lucky and roll it twice or not at all.
I had a colleague who was fond of saying that “I am 90% confident that we will complete the project on time”. When he would make this statement, I would ask ‘if we were to execute this project exactly the same way 100 times, then we would only finish it on time 90 of those times?” Unfortunately, we can “not step twice into the same river”, nor can we execute the same project more than once.
Many variables are nearly impossible to measure that contribute to the outcome of a project. For example, team cohesion is (in my opinion) a primary contributor to project success. That is not a variable that we generally measure. Even if we could measure the variables we don’t know if we are capturing cause-effect or simply correlated relationships. When the rooster crows in the morning, he is not causing the sun to rise, but his crow is highly correlated with sunrise.
Interpreting Statistics
People don’t understand statistical language as evidenced in the 20% chance of rain interpretation. The CIA has stopped assigning percentages when describing their intelligence assessments. They have started using phrases such as “not likely”, “likely”, and “very likely” because terms are more understandable.
Project managers could use similar terms or scaled rating systems to more effectively discuss expected outcomes with their teams. The typical risk rating system of assessing impact and probability on a 1-3 scale is a good example using a relative rating that people can easily comprehend. If 3 is “very likely” and 1 is “not likely”, it is easy for the project teams to compare one event against another.
Statistics and probabilities are powerful and useful tools. In manufacturing-like processes, they can be used to drive quality, consistency, and predictive repeatability. When assessing large portfolios of projects, they can also provide valuable insights. But on an individual project they must be used with extreme caution, if at all.
Sources:
Statistics. (n.d.). Retrieved March 22, 2015, from http://en.wikipedia.org/wiki/Statistics
(n.d.). Retrieved March 22, 2015, from http://www.npr.org/series/333708682/risk-and-reason
Butterfly Effect. (n.d.). Retrieved March 22, 2015, from http://en.wikipedia.org/wiki/Butterfly_effect
(n.d.). Retrieved March 22, 2015, from https://www.pmi.org/~/media/PDF/Business-Solutions/PMI-Pulse Report-2013Mar4.ashx
Heraclitus. (n.d.). Retrieved March 31, 2015, from http://en.wikiquote.org/wiki/Heraclitus
© 2015, Project Management Essentials, LLC