Jorge Quitério Search
IT & Engineering

Statistical sampling in control testing

This article explains why 59 is a common sample size in control testing, what a sample without deviations demonstrates, and why population size has little effect on the required sample. A simulated control illustrates the operating characteristic of a sampling plan.

Every information security audit faces the same constraint. There are 10,000 access-review records, 3,000 change tickets and one auditor with a week. Not every item can be examined. The auditor takes a sample, inspects it and draws a conclusion about the whole population.

The inference from "these 59 tickets were compliant" to "the control operates effectively" is a statistical inference, whether or not the audit report states it. Done properly, it provides assurance; done poorly, it provides only anecdotal evidence.

This article addresses three questions: why auditors frequently use a sample of 59, what a sample without deviations demonstrates, and why auditing a population of 100,000 does not require a hundred times more work than auditing 1,000.

Several standards and papers address this problem.

ISO 19011:2018 gives general guidance on auditing management systems, and ISO/IEC 27007:2020 adapts it to information security management systems. Both treat sampling as a normal way of collecting audit evidence and distinguish two kinds:

  • judgemental sampling, where the auditor picks items based on experience;
  • statistical sampling, where items are chosen at random and the conclusion carries a stated confidence.

ISACA's IT Audit Framework (ITAF), whose 5th edition appeared in February 2026, makes the same distinction for IT audit and assurance work; its companion guideline 2208 is devoted to IT audit sampling. Judgemental samples are good at finding problems, because an experienced auditor knows where they are likely to occur. Only statistical samples support a quantified conclusion about the population.

The IIA's Three Lines Model (2020) places internal audit as the independent assurance provider to the governing body. The statistics come from Clopper and Pearson (1934), whose exact binomial bound remains the reference for an upper bound on a rate given the observed sample.

The control test as an estimation problem

Take a control that should operate every time: "every production change has an approved ticket". The auditor is estimating a rate, the share of changes where the control failed, called the deviation rate. Two numbers are fixed before anyone opens a ticket:

  • the tolerable deviation rate pTp_T, the worst failure rate at which the control still counts as effective (often 5%);
  • the confidence CC required in the conclusion (often 90% or 95%).

The question then becomes precise: how large must a random sample be so that, if it contains no deviations, it can be stated with confidence CC that the true rate is below pTp_T?

Sample size for zero deviations

If the true deviation rate were exactly pTp_T, the chance that nn random items all pass is (1−pT)n(1-p_T)^n. This probability should be at most 1−C1 - C, so that a sample without deviations would be unlikely if the control performed that poorly:

(1−pT)n≤1−C⟹n≥ln⁡(1−C)ln⁡(1−pT). (1-p_T)^n \le 1 - C \quad\Longrightarrow\quad n \ge \frac{\ln(1 - C)}{\ln(1 - p_T)} .

For pT=5%p_T = 5\% and C=95%C = 95\% this gives n=59n = 59, a widely used sample size in control testing. Other combinations:

Confidence Tolerable rate Sample, no deviations allowed
90% 5% 45
95% 5% 59
95% 10% 29
95% 2% 149
95% 1% 299

For small rates, n≈3/pTn \approx 3/p_T: the "rule of three". It is the same rule that governs AI test campaigns.

Effect of population size

Intuition suggests that a population ten times larger requires a sample ten times larger. This is not the case. Sampling without replacement from a finite population, the exact calculation uses the hypergeometric distribution, and for 95% confidence at a 5% tolerable rate it needs:

Population Sample
100 45
500 56
10,000 59

Beyond a few hundred items, the population size is almost irrelevant. The required sample size is determined by the desired precision and confidence rather than by the size of the population.

A cook tasting soup relies on the same fact. One spoonful indicates whether a small pan or a large pot needs more salt, provided the soup has been stirred. The size of the pot does not change the size of the spoon.

Samples with deviations

Deviations weaken the conclusion quickly. With 59 items and one deviation, the exact one-sided 95% upper bound on the rate is about 7.8%, above the 5% tolerance. With two, about 10.3%. To keep "below 5% at 95% confidence" while allowing one deviation, the sample must be 93 items; allowing two, 124.

So an auditor who expects some deviations should plan the larger sample at the start. Deciding the sample size after seeing the results changes the statistics of the test, unless the sampling plan said in advance how it would expand.

Conditions for validity

  1. Random selection. Every item must have a known chance of selection. "The tickets that were easy to find" proves nothing about the others, just as a spoonful from the surface of an unstirred pot says nothing about the bottom.
  2. A defined population. The conclusion covers only what was sampled. Changes made outside the ticketing system are not in the population at all, which is why completeness of the population is an audit test in its own right.
  3. A clear definition of a deviation, fixed before sampling.

The figure below simulates a control audit. A sample of 59 without deviations leaves at most a 5% chance of trusting a control that actually fails 5% of the time, and the same sample says much less if it contains even one deviation.

The grid is a population of 2,000 control executions; orange ones failed. The curve is the operating characteristic: for every possible true failure rate, the probability that the sampling plan leads the auditor to rely on the control.

Suggested settings:

  1. Read the curve. With 59 samples and 0 deviations allowed, follow the curve to the "tolerable 5%" line. It crosses at 4.8%, just under 5%. This is the meaning of "59 at 95% confidence".
  2. An effective control. Set the true rate to 1% and draw a few samples. Most come back clean and say "rely on it". Now set the true rate to 8% and draw again. Samples without deviations still occur, but only about once in 140 draws.
  3. Allowing one deviation. Allow 1 deviation with 59 samples. The curve shifts right, and the chance of trusting a 5%-failing control rises to about 20%. Now raise the sample to 93: it falls back under 5%.
  4. Effect of a single deviation. Find a sample with exactly one deviation. The "95% upper bound" readout says about 7.8%. A single deviation is therefore enough to prevent the conclusion that the control is within tolerance.

Derivation of 59. The chance of relying on the control is P(X≤c)P(X \le c) where X∼Binomial(n,p)X \sim \text{Binomial}(n, p) counts deviations. For c=0c = 0 that is (1−p)n(1-p)^n, and (0.95)59=e59ln⁡0.95=e−3.03≈0.048(0.95)^{59} = e^{59 \ln 0.95} = e^{-3.03} \approx 0.048. For larger cc the widget sums the binomial probabilities exactly, through the regularised incomplete beta function.

A sample of 59 items without deviations supports a precise statement. It bounds the failure rate below 5% with 95% confidence, whether the population has 1,000 or 100,000 items. One deviation in that sample already moves the bound to about 7.8%.

This gives auditors and control owners a shared language. A control owner can ask what evidence is required and receive a quantified answer. In practice:

  • State the tolerable rate and confidence in the audit plan, not after the fact.
  • Plan for expected deviations up front (93 items for one, 124 for two).
  • Test the completeness of the population before sampling from it.
  • Report the conclusion with its numbers, so the governing body can weigh it against its risk appetite. That is where the decision belongs, as I argued in the role of information security in risk decisions.

References

  1. ISO 19011:2018, Guidelines for auditing management systems.
  2. ISO/IEC 27007:2020, Information security, cybersecurity and privacy protection - Guidelines for information security management systems auditing.
  3. ISACA, IT Audit Framework (ITAF): A Professional Practices Framework for IT Audit, 5th ed., 2026, with Companion Performance Guidelines 2208, Information Technology Audit Sampling.
  4. The Institute of Internal Auditors, The IIA's Three Lines Model: An Update of the Three Lines of Defense, 2020.
  5. C. J. Clopper and E. S. Pearson, "The use of confidence or fiducial limits illustrated in the case of the binomial", Biometrika, 26(4), 404-413, 1934.