Measurement in AI governance
ISO/IEC 42001, the NIST AI RMF and the EU AI Act all require evidence that AI systems behave as intended. Evidence about a statistical system is itself statistical, and a simulation of test campaigns shows how much testing a performance claim requires.
Suppose a vendor states that its model "makes errors in fewer than 1% of cases", based on 50 test examples, all answered correctly.
This evidence is weak. Fifty tests without errors are consistent with an error rate of 5%, five times the claim. This article explains why and simulates the corresponding test campaigns.
AI governance is often discussed as a matter of principles, policies and committees. These are important, but every serious framework eventually requires evidence that a system does what its owners claim, and evidence about the behaviour of a statistical system is itself statistical. This article examines how much testing an AI performance claim requires.
The relevant standards and literature are as follows.
ISO/IEC 42001:2023 specifies an AI management system. It uses the same harmonised clause structure as ISO/IEC 27001, from context and leadership (clauses 4 and 5) to internal audit, management review and improvement (clauses 9 and 10). Its Annex A lists 38 controls in nine areas, from AI policy to data, the AI system life cycle and third-party relationships. Two elements have no direct equivalent in an information security management system: an AI system impact assessment (clause 6.1.4) and controls on the provenance and quality of data. ISO/IEC 23894:2023 gives guidance on AI risk management and ISO/IEC 42005:2025 on conducting the impact assessment.
The NIST AI Risk Management Framework 1.0 (January 2023) is voluntary and organised around four functions: Govern, Map, Measure and Manage. This article concerns the third.
The EU AI Act, Regulation (EU) 2024/1689, is law. Its timetable was amended by Regulation (EU) 2026/1744, the Digital Omnibus on AI, in force since 27 July 2026:
| Obligation | Applies from |
|---|---|
| Prohibited practices | 2 February 2025 |
| General-purpose AI models | 2 August 2025 |
| Transparency obligations (Article 50) | 2 August 2026 |
| High-risk systems listed in Annex III | 2 December 2027 (previously 2 August 2026) |
| High-risk AI in products under Annex I | 2 August 2028 (previously 2 August 2027) |
The omnibus also softened the AI literacy obligation: providers and deployers must now take measures to support AI literacy rather than ensure a given level of it. Transitional details vary by provision, so the consolidated text on EUR-Lex should be checked before relying on a specific date.
The statistics behind this article are old and reliable: the exact binomial confidence bound of Clopper and Pearson (1934) and its convenient shortcut, the rule of three, discussed by Jovanovic and Levy (1997).
Integration with an existing management system
For an organisation that already runs ISO/IEC 27001, the practical advice is to extend that system rather than build a second one. Internal audit, management review, document control, supplier management and incident handling can serve both. What is genuinely new is the impact assessment, the data controls, and the need to measure the behaviour of the systems themselves.
Interpreting a test run without errors
Suppose the true error rate is and independent, representative test cases are run. The chance of seeing no errors at all is . The largest error rate still consistent with a run without errors, at 95% confidence, is the that makes this chance exactly 5%:
That is the rule of three. It explains the opening example: 50 clean tests only show that (exactly 5.8%). The same arithmetic applies to everyday claims: a commuter who has taken a new train connection ten times without a delay has evidence only that delays affect fewer than about 30% of journeys. To support "below 1%" with zero errors, 299 test cases are needed. Allowing one error raises that to 473; allowing two, to 628.
Error rates by group
Impact assessments ask about effects on particular groups. Each group needs its own evidence. If a claim must hold for each of five user groups, the 299 cases are needed per group, and the smallest group sets the pace. An aggregate accuracy figure can hide a subgroup with a few dozen test cases and therefore almost no information. Face recognition is the best-known case: NIST's 2019 evaluation of demographic effects found that, for many algorithms, false-match rates differed between demographic groups by factors of ten or more, differences that an overall accuracy figure did not show.
Independence and representativeness matter as much as the count. Test cases copied from the training data, or near-duplicates of each other, carry less information than their number suggests.
Monitoring is measurement too
A validated system can drift as its inputs change. A common monitoring statistic, the population stability index, compares the distribution of a feature in production () with its distribution at validation ():
the sum of the two Kullback-Leibler divergences, which I explored in Kullback-Leibler divergence and Hellinger distance. Thresholds such as 0.1 and 0.25 are industry conventions rather than statistical tests, and should be treated as triggers for investigation.
The figure below simulates test campaigns. A 95% upper bound computed from a test campaign is above the true error rate at least 95% of the time, and zero failures in tests bounds the rate at about .
Suggested settings:
- The vendor example. Set the true error rate to 5% and the campaign to 50 tests. Over several single campaigns, zero or one error occurs fairly often, and the bound remains above 5%: the test cannot distinguish a 5% system from a perfect one.
- Coverage. Press "Run 1,000 campaigns". The readout "campaigns whose bound covers the truth" stays at or above 95% for all settings. That is the definition of a 95% confidence bound, verified by simulation.
- Required sample size. Set the true rate to 0.1% (a genuinely good system) and the claim to 1%. The number of tests at which most campaigns support the claim is close to 300.
- Stricter claim. Lower the claim to 0.2% with the same system. The number of tests needed increases sharply, because .
Derivation of the rule of three. Taking logarithms of gives . For small , , so and . For campaigns with errors, the widget computes the exact Clopper-Pearson bound by solving numerically.
AI governance frameworks ask for evidence, and evidence has a sample size. A performance claim without its test count and confidence level cannot be verified; a claim such as "error rate below 1% (95% confidence, 0 errors in 300 independent test cases)" can. In practice:
- Require every claim to state its sample size and confidence level.
- Size the evidence for the smallest subgroup that matters.
- Keep the test set representative of deployment and separate from training data.
- Tie monitoring thresholds to actions, inside the same management system as every other control.
None of this requires new institutions. It requires governance to treat measurement as a control like any other.
References
- ISO/IEC 42001:2023, Information technology - Artificial intelligence - Management system.
- ISO/IEC 23894:2023, Information technology - Artificial intelligence - Guidance on risk management.
- ISO/IEC 42005:2025, Information technology - Artificial intelligence - AI system impact assessment.
- NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 of 8 July 2026 (Digital Omnibus on AI), Official Journal of the European Union.
- C. J. Clopper and E. S. Pearson, "The use of confidence or fiducial limits illustrated in the case of the binomial", Biometrika, 26(4), 404-413, 1934.
- B. D. Jovanovic and P. S. Levy, "A look at the rule of three", The American Statistician, 51(2), 137-139, 1997.