Performance measures derived from strategy
Technical teams rarely lack metrics, but they often lack the few measures that connect their work to organisational objectives. A small economic model shows the consequences of rewarding only what is easy to count.
Engineering and security teams can produce dashboards of almost anything: tickets closed, vulnerabilities found, pipelines run, hours logged. Most of those numbers are easy to count and hard to act on. Some are counterproductive.
Familiar examples illustrate this. A help desk rewarded for closing tickets quickly closes them before resolving them. A team measured on story points inflates its estimates. A security programme reports "vulnerabilities found", a figure that rises both when scanning improves and when the software deteriorates.
Education provides a familiar case outside technology. A school judged mainly on the share of pupils who pass an exam has an incentive to concentrate teaching on pupils just below the pass mark, at the expense of both the strongest and the weakest, a pattern documented in research on school accountability.
This article is about choosing the few measures that reflect strategy and setting the rest aside. It examines how a team's behaviour changes when only what is easy to measure is rewarded, and how paired measures correct this.
The relevant literature is as follows.
The balanced scorecard. Kaplan and Norton (1992) proposed looking at an organisation from four perspectives (financial, customer, internal processes, learning and growth) and choosing a few linked measures for each objective.
Delivery measures. The DORA research programme and the book Accelerate (Forsgren, Humble and Kim, 2018) identified four software delivery measures. They pair speed (deployment frequency, lead time) with stability (change failure rate, time to restore).
Gaming. Goodhart's law, usually paraphrased as "when a measure becomes a target, it ceases to be a good measure", comes from Charles Goodhart's 1975 work on monetary policy; Campbell (1979) stated a similar law for social indicators.
The economics. Holmström and Milgrom (1991) showed formally that when a job has several tasks and only some are measured, incentives on the measured ones pull effort away from the unmeasured ones. Their multitask principal-agent model is the basis of the figure in this article.
Starting from objectives
A measure answers a question. If the objective is to deliver reliable software faster, the measures should show speed and stability of delivery, not story points completed.
Three kinds of measure
- Key performance indicators (KPIs) show progress towards an objective.
- Key risk indicators (KRIs) warn that an objective is in danger.
- Activity metrics describe work done: useful for running a team, misleading when presented as outcomes.
"Number of vulnerabilities found" is an activity metric. "Share of critical vulnerabilities remediated within the agreed time" is a KPI: it moves only when the outcome does.
Example: a platform and security team
| Objective | Measure | Type |
|---|---|---|
| Deliver changes quickly | Lead time from commit to production | KPI |
| Deliver changes safely | Change failure rate; time to restore service | KPI |
| Keep exposure low | Critical vulnerabilities remediated within SLA | KPI |
| Detect drift early | Share of services not on the supported platform baseline | KRI |
| Build capability | Share of teams using the standard pipeline without assistance | KPI |
Principles for measure design
- Few measures. If a team cannot name its measures from memory, it has too many.
- Pair them. Every speed measure needs a quality measure beside it.
- Expect gaming. Review measures regularly and retire those that no longer reflect the objective.
- Show trends, not snapshots.
- Discuss before judging. A measure opens a conversation; it is not a verdict.
The model below illustrates Holmström and Milgrom's result: if only one of two necessary tasks is rewarded, a rational team neglects the other, and real value can fall to zero even while the measured score is at its best. Pairing the measures restores value.
In the model, a team splits effort between speed and quality to do as well as possible on the target it is given. Effort is costly, and the two kinds compete for the same hours. Real value needs both: here it is , so either one at zero means no value.
- Speed only. Press "Only speed is measured". The target score is high, quality effort falls to zero, and real value is zero. This is Goodhart's law in its simplest form.
- Paired measures. Press "Pair speed with quality". Effort splits, the score is lower, and real value reaches its maximum. The dashboard figure is lower, but the outcome is better.
- Unbalanced target. Set the weight to 80% speed and 20% quality, then raise the competition for time from 0 to 0.9. Quality effort shrinks and then disappears altogether once the competition passes 0.25: the more two tasks compete for the same time, the greater the risk of under-weighting one of them.
Derivation. The team chooses efforts to maximise , where and are the weights in the target and measures competition for time. Setting the derivatives to zero gives and , truncated at zero. With , would be negative, so it is zero: quality gets no effort at all. With equal weights the efforts are equal, and reaches its maximum.
Choosing measures is a statement of priorities. People pay attention to what is measured and discussed, often more than to anything written in a strategy document. That makes measurement a leadership decision rather than a reporting task. Measures should be few, every speed measure should be paired with a quality measure, and the multitask model should be kept in mind: whatever is left unmeasured is implicitly deprioritised.
References
- C. A. E. Goodhart, "Problems of monetary management: the U.K. experience", in Papers in Monetary Economics, Reserve Bank of Australia, 1975.
- D. T. Campbell, "Assessing the impact of planned social change", Evaluation and Program Planning, 2(1), 67-90, 1979.
- B. Holmström and P. Milgrom, "Multitask principal-agent analyses: incentive contracts, asset ownership, and job design", Journal of Law, Economics, & Organization, 7 (special issue), 24-52, 1991.
- R. S. Kaplan and D. P. Norton, "The balanced scorecard: measures that drive performance", Harvard Business Review, 70(1), 71-79, 1992.
- N. Forsgren, J. Humble and G. Kim, Accelerate: The Science of Lean Software and DevOps, IT Revolution, 2018.