Jorge Quitério Search
IT & Engineering

Risk matrices and loss exceedance curves

A five-by-five risk matrix can place two risks in the same cell when one is fifty times more likely to cause a loss of a million euros in a year. A simulation of both risks shows what a loss exceedance curve tells a risk owner that a colour cannot.

Most information security risk registers end in a matrix: likelihood on one axis, impact on the other, five levels each, and a colour in every cell. It is easy to explain and to complete, and it is widely used in board reporting.

It also discards most of what the analyst knows, and sometimes ranks risks in the wrong order. In 2008 Louis Anthony Cox Jr. set out the problems in a paper titled "What's wrong with risk matrices?".

This article illustrates the main problem with a simulation: two risks that land in the same amber cell, one of which is about fifty times more likely to produce a million-euro year. Then it shows the alternative, the loss exceedance curve, and how a risk appetite can be written on it.

Several standards and papers address this problem.

Cox (2008) showed that risk matrices have poor resolution, can assign higher ratings to quantitatively smaller risks, depend on ambiguous inputs, and give misleading guidance on where to spend mitigation money.

Budescu, Broomell and Por (2009) studied how people read the probability words used in IPCC reports, such as "likely" and "very likely", and found wide disagreement between readers on the numbers behind them, even with guidance at hand.

Open FAIR, maintained by The Open Group in its Risk Taxonomy (O-RT) and Risk Analysis (O-RA) standards, decomposes risk into the frequency of loss events and the magnitude of each loss, each estimated as a range.

Hubbard and Seiersen (2nd edition, 2023) make the practical case for quantifying cybersecurity risk with calibrated expert estimates and Monte Carlo simulation, rather than waiting for perfect data.

ISO/IEC 27005:2022 accepts quantitative as well as qualitative risk analysis.

Two risks in one cell

Consider two risks, each assessed as possible (an event in roughly three years out of ten) and high impact (a typical loss of about €200,000). Both land in the same cell.

Now describe them as distributions. In both, events arrive as a Poisson process at 0.3 per year, and each loss is lognormal with a median of €200,000. They differ only in the spread:

  • Risk A is well understood: 90% of its events cost between about €90,000 and €450,000.
  • Risk B is not: 90% of its events cost between about €17,000 and €2.4 million.
Risk A Risk B
Probability of at least one event in a year 26% 26%
Expected annual loss about €68,000 about €185,000
Probability of losing more than €1 million in a year about 0.08% about 4.4%
99th percentile of annual loss about €0.6 million about €3.2 million

Both have the same likelihood, typical impact and colour, yet B is more than fifty times as likely to produce a million-euro year, and its expected loss is almost three times larger. The difference lies entirely in the tail, which the matrix cannot represent.

Household insurance has the same structure. Two homes in the same street may face the same chance of a burglary and the same typical loss, but if one of them holds a valuable collection, its worst burglary costs many times more. An insurer that priced both policies from the typical loss would undercharge the second household; insurers price the tail, which is what the loss exceedance curve below shows. The same point is made in the γ-order generalized normal distribution: when outcomes are heavy-tailed, the typical value says little about the risk.

Further limitations

Each of Cox's problems follows from rounding continuous quantities into a few categories:

  • Poor resolution: risks that differ by orders of magnitude share a cell.
  • Ranking errors: banding can rate a smaller risk above a larger one, particularly when frequency and severity are negatively correlated, as they often are: frequent incidents tend to be small.
  • Ambiguous inputs: "possible" and "major" mean different numbers to different people.
  • Misleading allocation: colours do not say where an extra euro of mitigation removes the most loss.

Loss exceedance curves

The output of a quantitative analysis is a loss exceedance curve: for each amount xx, the probability that annual loss exceeds xx,

Fˉ(x)=P(L>x),L=∑k=1NXk, \bar F(x) = P(L > x), \qquad L = \sum_{k=1}^{N} X_k ,

with NN the number of events in a year and XkX_k their losses. It answers the questions risk owners ask: how likely is a bad year, and how bad is a very bad one?

Risk appetite can be drawn on the same axes: "no more than a 1% chance of losing more than €2 million in a year". A risk is within appetite if its curve passes below that point (or below a whole appetite curve). Mitigations are compared by how far they push the curve down, which can be set against their cost.

Quantification under uncertainty

A common objection is that the inputs are too uncertain to quantify. A matrix, however, does not remove uncertainty; it conceals it in verbal categories. A range like "between €50,000 and €2 million, 90% confidence" states the uncertainty and lets the arithmetic carry it through. If the resulting curve is wide, this is shown explicitly, and the value of reducing that uncertainty can itself be estimated, in the spirit of the value of the stochastic solution.

It also keeps the roles distinct: security supplies the distributions; the risk owner decides whether the curve is acceptable, as in the role of information security in risk decisions.

The figure below simulates both risks. Two risks with the same frequency and the same median loss can have completely different chances of a very bad year, and only the curve shows it.

Each risk is simulated for 20,000 years. The black dot represents the risk appetite: at most a 1% chance of exceeding the amount on the slider.

Suggested settings:

  1. Default settings. With the defaults, the curve for A has almost vanished by €1 million, while the curve for B is still around 4-5%. The two risks share a cell but have very different tails.
  2. Risk appetite. With an appetite of €2 million at 1%, A is within appetite and B is not. B falls within appetite only when the threshold is raised to a 1% chance of losing around €3.4 million.
  3. Reduced uncertainty for B. Lower B's spread to 0.5. The two curves coincide: without the additional spread, the matrix classification is accurate. The spread, rather than the label, determines the risk.
  4. Higher frequency. Raise the event rate to 2 per year. Both curves move right and the probability of a bad year rises sharply, while the cell (if only "possible" were changed to "likely") moves one box.

The closed forms. The expected annual loss of a compound Poisson process is E[L]=λ E[X]E[L] = \lambda\, E[X]. For a lognormal loss with median mm and log-spread σ\sigma, E[X]=m eσ2/2E[X] = m\, e^{\sigma^2/2}. So

E[LA]=0.3×200,000×e0.125≈68,000,E[LB]=0.3×200,000×e1.125≈185,000. E[L_A] = 0.3 \times 200{,}000 \times e^{0.125} \approx 68{,}000, \qquad E[L_B] = 0.3 \times 200{,}000 \times e^{1.125} \approx 185{,}000 .

The probability of at least one event is 1−e−λ=1−e−0.3≈26%1 - e^{-\lambda} = 1 - e^{-0.3} \approx 26\% for both. The tail differs because a single event exceeds €1 million with probability 1−Φ(ln⁡5/σ)1 - \Phi(\ln 5/\sigma), which is about 0.07% for σ=0.5\sigma = 0.5 and about 14% for σ=1.5\sigma = 1.5. The simulation's estimates of expected loss vary from run to run for risk B, because heavy tails make averages converge slowly.

A matrix compresses a distribution into a colour, and the tail is the first thing lost. A pragmatic approach:

  • Keep the matrix for communication if people are used to it, but derive its cells from quantities rather than the other way round.
  • Quantify the top ten risks with ranges for frequency and loss and a few thousand Monte Carlo draws. A spreadsheet or a short script is enough.
  • Ask assessors for a 90% interval instead of an "impact level".
  • Write risk appetite as exceedance probabilities, so the governing body decides on numbers it can compare.

References

  1. L. A. Cox Jr., "What's wrong with risk matrices?", Risk Analysis, 28(2), 497-512, 2008.
  2. D. V. Budescu, S. Broomell and H.-H. Por, "Improving communication of uncertainty in the reports of the Intergovernmental Panel on Climate Change", Psychological Science, 20(3), 299-308, 2009.
  3. The Open Group, Risk Taxonomy (O-RT) and Risk Analysis (O-RA) standards (Open FAIR).
  4. D. W. Hubbard and R. Seiersen, How to Measure Anything in Cybersecurity Risk, 2nd ed., Wiley, 2023.
  5. ISO/IEC 27005:2022, Information security, cybersecurity and privacy protection - Guidance on managing information security risks.