Jorge Quitério Search
Mathematics

Sequential Bayesian updating of a failure rate

A prior belief about a failure rate is updated with test results to give a posterior belief. With a conjugate prior the update consists of two additions, the posterior narrows approximately as 1/√n, and it can be compared directly with the classical Clopper-Pearson bound.

Before a control is tested, some prior knowledge about it already exists. The vendor's documentation, last year's audit and the fact that it has never caused an incident all count. A purely classical analysis disregards this knowledge. A Bayesian analysis states it as a distribution and updates it with each test result.

The prior is subjective, since it is a stated assumption. It is, however, auditable, which an unstated judgement is not. For the most common question in assurance, the failure rate of a control, the Bayesian calculation reduces to two additions.

Thomas Bayes' essay, published posthumously in 1763, addressed this problem: inferring the probability of an event from the number of times it happened in a number of trials, starting from a uniform prior. Laplace (1774) reached the same result independently and derived the "rule of succession". Jeffreys (1946) proposed priors that do not depend on how a parameter is labelled, which for a proportion gives the Beta(½, ½) prior. Brown, Cai and DasGupta (2001) compared interval estimates for a proportion and found that the Bayesian interval under the Jeffreys prior has excellent frequentist coverage, better than the textbook normal approximation. Gelman and colleagues' Bayesian Data Analysis (2013) is the standard modern reference.

The beta-binomial model

Let pp be the failure rate of a control. Describe the prior belief as a Beta distribution,

π(p)∝pa−1(1−p)b−1, \pi(p) \propto p^{a-1} (1-p)^{b-1},

which can be read as "a−1a - 1 failures and b−1b - 1 successes' worth of previous experience". Now run nn independent tests and observe kk failures. The likelihood is pk(1−p)n−kp^k (1-p)^{n-k}, and multiplying gives

π(p∣k,n)∝pa+k−1(1−p)b+n−k−1, \pi(p \mid k, n) \propto p^{a+k-1}(1-p)^{b+n-k-1} ,

which is again a Beta distribution: Beta(a+ka + k, b+n−kb + n - k). The failures are added to aa and the successes to bb. The posterior mean is (a+k)/(a+b+n)(a + k)/(a + b + n), a weighted average of the prior mean and the observed rate, with weights proportional to the prior's "pseudo-tests" and the real tests.

The same arithmetic applies when choosing between two sellers on an online marketplace. One has 3 positive ratings out of 3, the other 290 out of 300. With a uniform prior the posterior means of their rate of satisfied customers are (1+3)/(2+3)=0.80(1 + 3)/(2 + 3) = 0.80 and (1+290)/(2+300)≈0.96(1 + 290)/(2 + 300) \approx 0.96, so the second seller is the better choice despite the lower raw score: three ratings are too few to outweigh the prior. This is Laplace's rule of succession, and the same weighting protects an auditor from over-interpreting a handful of clean tests.

Choosing a prior

  • Uniform, Beta(1, 1): every rate equally plausible. Bayes' own choice.
  • Jeffreys, Beta(½, ½): the reference prior; slightly favours rates near 0 and 1.
  • Optimist, Beta(1, 30): "we think failures are rare", worth about 30 clean tests.
  • Pessimist, Beta(5, 20): "we think about one in five fails", worth about 25 tests.

With sufficient data the choice has little effect: the posterior is dominated by the likelihood. With little data it has a large effect, which is when it should be stated explicitly. For the seller with 290 good ratings out of 300, the four priors in this list give posterior satisfaction rates between 0.95 and 0.97; for the seller with 3 out of 3, the same priors give anything from 0.80 to 0.97.

Comparison with the Clopper-Pearson bound

The auditor's classical answer, from statistical sampling in control testing, is the one-sided Clopper-Pearson upper bound: after 59 clean tests, the rate is below about 5% with 95% confidence. The Bayesian equivalent under a uniform prior is the 95th percentile of Beta(1, 60), which is 1−0.051/60≈4.9%1 - 0.05^{1/60} \approx 4.9\%. The two approaches give almost the same number. With a few failures the two bounds stay close, and the Bayesian one has the advantage of a direct reading: "given the tests and the prior, there is a 95% probability that the rate is below this value".

In the figure below, a prior is selected, a true failure rate is set, and tests are run.

Each ridge is the belief after one more batch, from the prior at the top to the latest at the bottom. The bullet chart compares the Bayesian and classical upper bounds with the tolerable rate.

Suggested settings:

  1. Fifty-nine clean tests. Choose the uniform prior and a true rate of 0.5%, then press "Test 50 more" and "Test 1 more" nine times. If no failure appears, the Bayesian 95% upper bound is about 4.9% and the Clopper-Pearson bound about 4.95%: nearly identical.
  2. Narrowing posterior. Keep testing in batches of 50. The ridges narrow steadily; their width falls roughly like 1/n1/\sqrt n.
  3. Optimistic prior. Choose the optimist prior and set the true rate to 15%. Initially the posterior gives little weight to the failures; after about a hundred tests the data dominate and the posterior moves to the true value.
  4. Effect of the prior. With only 10 tests, switch between priors without changing the data. The posteriors differ visibly: with little data the prior has a large influence, and it should be documented and justified.

Conjugacy and the sample size of 59. Bayes' theorem says posterior ∝ prior × likelihood. The prior kernel pa−1(1−p)b−1p^{a-1}(1-p)^{b-1} times the binomial kernel pk(1−p)n−kp^{k}(1-p)^{n-k} is pa+k−1(1−p)b+n−k−1p^{a+k-1}(1-p)^{b+n-k-1}, the kernel of Beta(a+ka + k, b+n−kb + n - k); the normalising constants follow automatically. With a uniform prior and nn clean tests the posterior is Beta(1, n+1n + 1), whose distribution function is 1−(1−p)n+11 - (1 - p)^{n+1}. Setting it to 0.95 gives the 95% upper bound 1−0.051/(n+1)1 - 0.05^{1/(n+1)}, which for n=59n = 59 is 4.87%. The classical bound solves (1−p)n=0.05(1-p)^{n} = 0.05 instead, giving 4.95%: one pseudo-test of difference.

Bayesian updating suits assurance work because it accumulates. This year's posterior is next year's prior, so evidence from previous audits, continuous control monitoring and incident history combines into one statement about a control, with its uncertainty attached. The same idea extends from proportions to rates, in Bayesian incident rates.

References

  1. T. Bayes, "An essay towards solving a problem in the doctrine of chances", Philosophical Transactions of the Royal Society of London, 53, 370-418, 1763.
  2. P.-S. Laplace, "Mémoire sur la probabilité des causes par les événements", Mémoires de l'Académie royale des sciences de Paris, 6, 621-656, 1774.
  3. H. Jeffreys, "An invariant form for the prior probability in estimation problems", Proceedings of the Royal Society A, 186(1007), 453-461, 1946.
  4. L. D. Brown, T. T. Cai and A. DasGupta, "Interval estimation for a binomial proportion", Statistical Science, 16(2), 101-133, 2001.
  5. A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari and D. B. Rubin, Bayesian Data Analysis, 3rd ed., CRC Press, 2013.