|Up a level|
Our research documents that projects across industries and geographies struggle to meet the most basic targets. Nine out of ten transport projects, six out of ten energy projects, seven out of ten dams, five out of ten technology projects, and ten out of ten Olympics do not meet their cost targets. Most strikingly, this trend has been constant, with no improvement over the past century. Why?
A brisk building boom of hydropower mega-dams is underway from China to Brazil. Whether benefits of new dams will outweigh costs remains unresolved despite contentious debates. We investigate this question with the “outside view” or “reference class forecasting” based on literature on decision-making under uncertainty in psychology. We find overwhelming evidence that budgets are systematically biased below actual costs of large hydropower dams—excluding inflation, substantial debt servicing, environmental, and social costs. Using the largest and most reliable reference data of its kind and multilevel statistical techniques applied to large dams for the first time, we were successful in fitting parsimonious models to predict cost and schedule overruns. The outside view suggests that in most countries large hydropower dams will be too costly in absolute terms and take too long to build to deliver a positive risk-adjusted return unless suitable risk management measures outlined in this paper can be affordably provided. Policymakers, particularly in developing countries, are advised to prefer agile energy alternatives that can be built over shorter time horizons to energy megaprojects.
In this issue of JIT Drummond argues that the fallacy of risk registers is caused by the fog of risk and the surplus reality of risk registers. This challenge argues that the shortcomings of are better described by the philosophic fallacies of false certainty and misplaced concreteness than by organizational rhetoric.
Out-of-control information technology (IT) projects have ended the careers of top managers, such as EADS CEO Noël Forgeard and Levi Strauss’ CIO David Bergen. Moreover, IT projects have brought down whole companies, like Kmart in the US and Auto Windscreen in the UK. Software and other IT is now such an integral part of most business processes and products that CEOs must know their IT risks, which are typically substantial – and overlooked. The analysis of a sample of 1,471 IT projects showed that the average cost overrun was 27% — but that figure masks a far more alarming “fat tail” risk. Fully one in six of the projects in the sample was a Black Swan, with a cost overrun of 200%, on average, and a schedule overrun of almost 70%. This highlights the true pitfall of IT change initiatives: It’s not that they’re particularly prone to high cost overruns on average – it is that there are a disproportionate number of Black Swans. By focusing on averages instead of the more damaging outliers, most managers and consultants have been missing the real risk in doing IT. In conclusion, the article outlines ideas as to what can be done to avoid Black Swans.
In this paper we characterise the propensity of big capital investments to systematically deliver poor outcomes as "fragility," a notion suggested by Nassim Taleb. A thing or system that is easily harmed by randomness is fragile. We argue that, contrary to their appearance, big capital investments break easily — i.e. deliver negative net present value — due to various sources of uncertainty that impact them during their long gestation, implementation, and operation periods. We do not refute the existence of economies of scale and scope. Instead we argue that big capital investments have a disproportionate (non-linear) exposure to uncertainties that deliver poor or negative returns above and beyond their economies of scale and scope. We further argue that to succeed, leaders of capital projects need to carefully consider where scaling pays off and where it does not. To automatically assume that "bigger is better," which is common in megaproject management, is a recipe for failure.
Implementing large-scale information and communication technology (IT) projects carries large risks and easily might disrupt operations, waste taxpayers’ money, and create negative publicity. Because of the high risks it is important that government leaders manage the attendant risks. We analysed the based on a sample of 1,355 public-sector IT projects. The sample included large-scale projects, on average the actual expenditure was $130 million and the average duration was 35 months. Our findings showed that the typical project had no cost overruns and took on average 24% longer than initially expected. However, comparing the risk distribution with the normative model of a thin tailed distribution, projects’ actual costs should fall within -30% and +25% of the budget in nearly 99 out of 100 projects. The data showed, however, that a staggering 18% of all projects are outliers with cost overruns >25%. Tests showed that the risk of outliers is even higher for standard software (24%) as well as in certain project types, e.g., data management (41%), office management (23%), eGovernment (21%) and management information systems (20%). Analysis showed also that projects duration adds risk: every additional year of project duration increases the average cost risk by 4.2 percentage points. Lastly, we suggest four solutions that public-sector organization can take: (1) benchmark your organization to know where you are (2) de-bias your IT project decision-making, (3) reduce the complexities of your IT projects, and (4) develop Masterbuilders to learn from the best in the field.
Given that Olympic Games held over the past decade each have cost USD 8.9 billion on average, the size and financial risks of the Games warrant study. The objectives of the Oxford Olympics study are to (1) establish the actual out turn costs of previous Olympic Games in a manner where cost can consistently be compared across Games; (2) establish cost overruns for previous Games, i.e., the degree to which final out turn costs reflect projected budgets at the bid stage, again in a way that allows comparison across Games; (3) test whether the Olympic Games Knowledge Management Program has reduced cost risk for the Games, and, finally, (4) benchmark cost and cost overrun for the Rio 2016 Olympics against previous Games. The main contribution of the Oxford study is to establish a phenomenology of cost and cost overrun at the Olympics, which allows consistent and systematic comparison across Games. This has not been done before.
Main findings of the study are, first, that average actual out turn cost for Summer Games is USD 5.2 billion (2015 level), and USD 3.1 billion for Winter Games. The most costly Summer Games to date are London 2012 at USD 15 billion; the most costly Winter Games Sochi 2014 at USD 21.9 billion. The numbers cover the period 1960-2016 and include only sports-related costs, i.e., wider capital costs for general infrastructure, which are often larger than sports-related costs, have been excluded.
Second, at 156 percent in real terms, the Olympics have the highest average cost overrun of any type of mega-project. Moreover, cost overrun is found in all Games, without exception; for no other type of mega-project is this the case. 47 percent of Games have cost overruns above 100 percent. The largest cost overrun for Summer Games was found for Montreal 1976 at 720 percent, followed by Barcelona 1992 at 266 percent. For Winter Games the largest cost overrun was 324 percent for Lake Placid 1980, followed by Sochi 2014 at 289 percent.
Third, the Olympic Games Knowledge Management Program appears to be successful in reducing cost risk for the Games. The difference in cost overrun before (166 percent) and after (51 percent) the program began is statistically significant.
Fourth, and finally, the Rio 2016 Games, at a cost of USD 4.6 billion, appear to be on track to reverse the high expenditures of London 2012 and Sochi 2014 and deliver a Summer Games at the median cost for such Games. The cost overrun for Rio – at 51 percent in real terms, or USD 1.6 billion – is the same as the median cost overrun for other Games since 1999.
Given the above results, for a city and nation to decide to stage the Olympic Games is to decide to take on one of the most costly and financially most risky type of mega-project that exists, something that many cities and nations have learned to their peril.
This essay briefly shines light on the current, normative, and positivist philosophies of how risk is debated in the field of project and program management. Then an alternative view is offered - Social Theories of Risk. This essay then analyses the National Programme for IT in the NHS (NPfIT), in its time the largest civilian ICT project in the world. The analysis focuses on the political struggles surrounding the project and asks the questions; What order was disturbed? Who disturbed the order? Who was blamed? and How was order re-established? The essay then contrasts notions from Social Theories of Risk with the idea of Governmentality to manage ICT project risk. The essay shows that Social Theories of Risk and Governmentality, both concepts not yet embraced in the ICT community, offer valuable insight into the study and the practice of ICT projects as clumsy solutions to wicked problems.
The Iron Triangle formulates the holy trinity of objectives of project management – cost, schedule, and benefits. As our previous research has shown, ICT projects deviate from their initial cost estimate by more than 10% in 8 out of 10 cases. Academic research has argued that Optimism Bias and Black Swan Blindness cause forecasts to fall short of actual costs. Firstly, optimism bias has been linked to effects of deception and delusion, which is caused by taking the inside-view and ignoring distributional information when making decisions. Secondly, we argued before that Black Swan Blindness makes decision-makers ignore outlying events even if decisions and judgements are based on the outside view. Using a sample of 1,471 ICT projects with a total value of USD 241 billion – we answer the question: Can we show the different effects of Normal Performance, Delusion, and Deception?
We calculated the cumulative distribution function (CDF) of (actual-forecast)⁄forecast. Our results show that the CDF changes at two tipping points – the first one transforms an exponential function into a Gaussian bell curve. The second tipping point transforms the bell curve into a power law distribution with the power of 2.
We argue that these results show that project performance up to the first tipping point is politically motivated and project performance above the second tipping point indicates that project managers and decision-makers are fooled by random outliers, because they are blind to thick tails. We then show that Black Swan ICT projects are a significant source of uncertainty to an organisation and that management needs to be aware of.
Finally, we draw implications about the underlying generative processes that lead to power law behaviour, which might help to further understand the pitfalls and shortcomings of cost and cost risk management in ICT projects.
The academic literature and popular press has chronicled large IT project failures for the last 40 years. Two points of contention surround this debate. First, quantitative studies found mixed support of a wide-spread crisis, questioning the representativeness of failure cases. Second, organizational theories disagreed on underlying assumptions about the nature of uncertainty, in particular about stability, locus of control, and controllability of the causes of IT project disasters. To advance the understanding of these two gaps four hypotheses were tested with a sample of 4,227 IT projects. The findings showed that outliers are stable phenomena following power laws, occurrence and impact of outliers differs between public and private sector, benefits management is associated with thinner tails and lower risk, and agile delivery methods do not statistically significantly influence the thickness of the tails. In sum, outliers are stable and non-random phenomena. They matter more than medians or means when it comes to IT project risk. Second, the notion of outliers bridges the gap between qualitative and quantitative studies. The findings also show that causes of outliers are, at least to some extent, internal and controllable by organizations. Lastly, the paper draws implications for organizational decision-making, learning, and risk management.