Forecast uncertainty for technological progress
If a technology's log cost is modelled as a random walk with drift, the forecast error has a known distribution: it grows with the horizon, and faster when the history is short. Hindcasting shows why most forecast intervals in strategy documents are too narrow.
Every technology strategy contains a forecast, usually as a single line: "battery pack cost will reach X by 2030". A single line conveys certainty but will almost certainly be wrong; the relevant question is by how much.
Doyne Farmer and François Lafond addressed this question in their paper "How predictable is technological progress?". Their answer is practical: if the yearly change in a technology's cost is treated as random around a steady trend, the width of the forecast error can be stated in advance for any horizon and any technology. This article examines how wide the interval around a technology forecast should be and why most are too narrow.
The consequences are concrete. Long-range energy projections in the 2010s repeatedly underestimated how quickly solar installations would grow, and plans for grids, subsidies and fossil-fuel plants were made on those projections. A household deciding whether to install solar panels now or in five years faces the same question on a small scale: the decision depends on how far panel prices will fall, and the honest answer is a range.
The background is the generalised Moore's law: many technologies get cheaper exponentially, at rates that differ from one technology to the next, as I discussed in Wright's law and Moore's law. Nagy and colleagues (2013) showed by hindcasting that such laws do forecast, with errors that can be measured. Farmer and Lafond (2016) formalised Moore's law as a correlated geometric random walk with drift, applied it to historical data on 53 technologies, derived a closed-form approximation for the distribution of forecast errors as a function of the horizon, and found that, properly scaled, the errors of all technologies at all horizons collapse onto the same distribution. They then made distributional forecasts for solar photovoltaic modules. Forecasting with error bars predates technology studies; the same logic sits behind the fans in my note on AR forecasts.
The model
Write for the log of unit cost. The simplest version of the model, without the moving-average correction Farmer and Lafond add, is
a random walk whose drift is the average yearly log-change (negative when costs fall). With years of history, the natural estimate of the drift is the average yearly change, , and the forecast years ahead is a straight line: .
Two sources of error
The forecast error has two parts. The shocks still to come add up to variance . And the drift itself is uncertain: has variance , and that error is multiplied by the horizon. Together,
The second term is often omitted in strategy documents. It grows with the square of the horizon, and it is large whenever the history is short compared with the horizon, which is the situation of every emerging technology.
The hindcasting simulation below illustrates the result: forecast errors of a random walk with drift have variance ; a fan that ignores the drift uncertainty covers far less than it promises when the history is short.
- Single forecast. With 10 years of history, the estimated improvement rate can easily miss the true 10% by five points or more. Pressing "Reveal the future" shows the actual path. With repeated "New history", the centre of the fan varies; this variation is the term.
- Hindcasting. Press "Hindcast 4,000 times". The dots (simulated error variance divided by ) sit on the blue curve , not on the dashed line . At 20 years with 10 years of history, the complete fan covers the truth about 92% of the time; the naive fan only about 73%.
- Longer history. Set the history to 40 years and hindcast again. The two curves converge (94% against 89%): with a long record, the drift is known well and only the future shocks matter.
- Emerging technology. Set the history to 5 years. The naive fan covers just over half the outcomes at 20 years. Even the complete fan falls short of 95% (about 87%), because itself is estimated from four changes; a distribution instead of the normal would repair most of that.
Derivation of the two terms. The true future is , and the estimate of the drift is . The forecast error is therefore . The two sums involve different shocks, so they are independent, and their variances are and . Adding them gives the formula.
When presenting a technology forecast, the rate should be reported with the length of its history, the interval should include both terms, and any narrow interval based on a short record should be treated with caution. A technology with ten years of data forecast twenty years out has a fan almost twice as wide as the naive one (). That width reflects what ten years of data can support. For the household considering solar panels, it means that a forecast based on a decade of prices should be read as a range of plausible prices, and the decision should still be acceptable at both ends of that range.
References
- B. Nagy, J. D. Farmer, Q. M. Bui and J. E. Trancik, "Statistical basis for predicting technological progress", PLoS ONE, 8(2), e52669, 2013.
- J. D. Farmer and F. Lafond, "How predictable is technological progress?", Research Policy, 45(3), 647-665, 2016.
- C. L. Magee, S. Basnet, J. L. Funk and C. L. Benson, "Quantitative empirical trends in technical performance", Technological Forecasting and Social Change, 104, 237-246, 2016.
- O. L. de Weck, Technology Roadmapping and Development: A Quantitative Approach to the Management of Technology, Springer, 2022.