Jorge Quitério Search
IT & Engineering

Assurance controls for AI-generated code

An AI coding assistant can be treated as a fast and confident contributor without accountability. The necessary controls already exist; the volume of code is what changes. A simulation of defects passing through layered controls shows why independent controls matter more than additional ones.

AI coding assistants now write a large share of new code in many teams. The debate about them tends to swing between "the code is excellent" and "the code is dangerous". Neither position addresses the main question.

A more useful model treats the assistant as a contributor who writes very quickly, has been trained on most public code, is always confident and carries no operational responsibility. Such a contributor would not be excluded, but their work would not be merged without review.

This article examines the controls needed for code produced at this volume, and why independent checks matter more than additional checks.

Anyone who has proofread their own writing knows the underlying problem. The author reads what was meant rather than what was written, and misses errors that a colleague sees at once. A second reading by the same author adds little; a reading by someone else adds a great deal. An assistant that writes both the code and its tests is an author proofreading its own text.

The relevant guidance comes from several standards and papers.

Secure development practice has always assumed that code can be wrong. NIST's Secure Software Development Framework (SP 800-218, 2022) and ISO/IEC 27001:2022 Annex A (notably A.8.28 on secure coding) describe the controls this article relies on.

Defence in depth has a well-known visual metaphor: James Reason's Swiss cheese model of accidents (1990, 2000). Each layer of defence has holes; an accident happens when the holes line up. Reason's point was organisational, but the arithmetic behind it is probability.

Common-cause failure is the reliability engineering term for holes that line up systematically: several protective layers that fail together because they share a cause. That is why redundancy in safety systems is designed to be diverse, and duplication alone is not enough.

Six core controls

  1. Own the tests. Tests are the specification. When the same assistant writes the code and its tests, the tests share the code's blind spots. Humans should define the tests for behaviour that matters; property-based tests are especially good at catching plausible-looking edge-case bugs.
  2. Review for intent. Generated code is tidy and idiomatic, which makes superficial review easy and misleading. Ask whether the change does what was intended and nothing else. Small changes make that answerable.
  3. Verify every dependency. Assistants sometimes suggest packages that do not exist, or whose names are one letter away from popular ones. Attackers register those names. Check, pin and scan.
  4. Keep secrets out of prompts. Use enterprise terms that exclude training on the organisation's data, and make the rules explicit for code, credentials and customer data.
  5. Record provenance. Mark AI-assisted changes (a commit trailer works) and keep the software bill of materials current.
  6. Keep a human accountable. The person who merges the change owns it.
Risk Control Reference
Incorrect or insecure logic Human-owned tests, review for intent, static analysis NIST SSDF PW.7, PW.8 · ISO/IEC 27001 A.8.28
Malicious or non-existent dependencies Registry verification, pinning, composition analysis NIST SSDF PW.4 · ISO/IEC 27001 A.5.21
Leakage of code or credentials Approved tools, data rules, secret scanning ISO/IEC 27001 A.5.14, A.8.12
Unclear origin of code Commit trailers, SBOM, signed commits NIST SSDF PS.3
Diffuse accountability Named owner for every merge NIST SSDF PO.2

Escape probability through layered controls

The controls act as successive filters. If tests, review and static analysis catch a given defect with probabilities p1,p2,p3p_1, p_2, p_3, and they fail independently, the probability that a defect slips through all three is

P(escape)=(1−p1)(1−p2)(1−p3). P(\text{escape}) = (1-p_1)(1-p_2)(1-p_3).

With catch rates of 60%, 50% and 40%, that is 0.4×0.5×0.6=12%0.4 \times 0.5 \times 0.6 = 12\%. Two consequences:

  • Volume. If AI doubles the code merged while the controls stay the same, the escape rate is unchanged and the number of escaped defects doubles. The cost of writing code has fallen; the cost of checking it has not.
  • Independence. The product formula needs independent layers. If the same model writes the code, its tests and its review, the layers share blind spots, and a defect that fools one tends to fool all.

The figure below illustrates this. Independent layers multiply their protection; correlated layers do not. In the fully correlated case, three layers provide no more protection than the best single layer.

  1. Default settings. About 12% of the 60 defects, roughly 7, reach the bottom. Over repeated runs the count varies around the value given by the formula.
  2. Doubled volume. Set 120 defects per sprint. The percentage is unchanged and the count doubles, which reflects the effect of AI assistance on volume.
  3. Shared blind spots. Switch on "shared blind spots". Now each defect has a single difficulty, and every layer catches it only if it is easy enough for that layer. The escape probability rises from 12% to 40%, the miss rate of the best layer.
  4. Improving one layer. With blind spots shared, raise review to 95%. Escapes fall to 5%. With independent layers, the same improvement matters much less. Correlated systems are only as strong as their strongest single layer, so that is where to invest.

Where the numbers come from. If layer ii misses with probability 1−pi1-p_i and misses are independent, the probability of missing at all three is the product of the miss probabilities. In the fully correlated model, each defect has a single difficulty UU, uniform between 0 and 1, and layer ii catches it when U<piU < p_i. A defect escapes all layers only if U≥piU \ge p_i for every ii, that is if U≥max⁡ipiU \ge \max_i p_i, which happens with probability 1−max⁡ipi1 - \max_i p_i.

AI-written code does not call for less trust than human-written code, but no code deserves more trust than the evidence supports. In practice this means applying the existing controls consistently and making them diverse: humans who own the tests, reviewers who assess intent, and tools that work differently from the model that wrote the code. Because writing code is now inexpensive, verification determines the pace of delivery. Teams that relied on the slow pace of manual coding as an informal quality gate no longer have that gate.

References

  1. NIST SP 800-218, Secure Software Development Framework (SSDF) Version 1.1, 2022.
  2. ISO/IEC 27001:2022, Information security management systems - Requirements, Annex A.
  3. J. Reason, Human Error, Cambridge University Press, 1990.
  4. J. Reason, "Human error: models and management", BMJ, 320(7237), 768-770, 2000.