Jorge Quitério Search
IT & Engineering

Security risks of autonomous coding agents

An assistant that suggests code is a productivity tool. An agent that runs commands, reads web content and pushes to repositories is a new identity in the environment. The article shows why a low per-page injection rate becomes a high daily probability, and which controls reduce the risk.

The first AI coding tools suggested text and waited for the user. The current generation takes actions. Coding agents open files, run shell commands, call APIs, read issue trackers and web pages, and open pull requests.

This is a qualitative change. An agent now holds credentials and takes actions, and its instructions can come from anything it reads. Any third party can publish text that an agent may later read. This article examines how serious the risk is when each attempt rarely succeeds, and which controls reduce it.

The relevant guidance comes from several standards and papers.

OWASP's Top 10 for Large Language Model Applications (2025 edition) lists prompt injection first and excessive agency among the top risks. NIST's AI Risk Management Framework (AI 100-1, 2023) and ISO/IEC 42001:2023 give governance structure. The underlying security principles are much older: Saltzer and Schroeder's design principles (1975), including least privilege, still describe most of the controls that work. The probability argument below is elementary, and the evidence argument uses the rule of three (Jovanovic and Levy, 1997).

Four risk categories

  1. Prompt injection through content. An agent cannot reliably tell instructions from its user apart from instructions embedded in what it processes. A README, an issue comment or a web page can contain text written to redirect it: run this command, add this dependency, send this file there. The situation resembles a new employee who acts on any note left on the desk, including one that asks for an urgent payment in the name of the chief executive.
  2. Excessive agency. An agent given a broad token to "fix the build" can usually also delete branches, change CI or publish packages. When something goes wrong, the damage is bounded only by what it was allowed to do.
  3. Secret exposure. Agents work where credentials live: environment variables, config files, SSH keys. Anything the agent can read, it can put in a request, a log or a commit.
  4. Supply chain through tools. Plugins and tool servers run with the agent's permissions. Each is a dependency.

Controls

Principle In practice
Least privilege Scope tokens to one repository and one task; read-only by default; no production credentials
Isolation Disposable sandboxes with an egress allowlist
Human approval for irreversible actions Merges, deployments, releases and deletions go through a person, typically as a pull request
Content is data Everything the agent reads is untrusted input; fetched text never changes its permissions
Short-lived secrets Credentials per session; no long-lived keys in the environment
Accountability and audit A distinct identity per agent, with logs of every command and API call
Vetted tools Pin and review plugins and tool servers like any other dependency

The pull request deserves emphasis: it is where an agent's work becomes a proposal a human accepts, with review, checks and history already built in. An agent that can push straight to the main branch bypasses a control that most teams already operate well.

Cumulative probability of injection

Suppose an injection works on one page in a thousand. Over nn independent exposures, the probability that at least one succeeds is

1−(1−p)n, 1 - (1 - p)^n ,

which for p=0.1%p = 0.1\% is 63% after 1,000 pages and 95% after 3,000. An agent that browses all day reaches those numbers quickly. Since pp cannot be reduced to zero, the factor that can be controlled is the impact of the one that gets through: expected loss is probability times impact, and least privilege shrinks the second factor.

Organisations already apply this reasoning to phishing. An employee who receives thousands of emails a year will eventually click on a convincing fraudulent one, however good the training. Security teams therefore limit what one click can do: multi-factor authentication, no administrator rights on everyday accounts, payment approvals by a second person. An agent reading thousands of pages needs the same limits.

The figure below illustrates two points. A success rate per page that sounds negligible becomes a near-certainty over a realistic workload, and a red-team exercise without successes provides less assurance than it appears to.

  1. Default settings. At 0.1% per page and 1,000 pages, the chance of at least one successful injection is 63%. Press "Simulate a day of browsing" several times and count the days with a hit.
  2. Improved defence. Improve the defence tenfold, to 0.01%. About 7,000 pages are then needed for even odds, a volume that a busy fleet of agents can reach in a day.
  3. Red-team results. Suppose a red team ran 300 attacks and none worked. The rule of three says the true rate could still be about 1% (3/300). Set pp to 1% and look at what 1,000 pages would mean.
  4. Even odds. The readout "pages until even odds" is ln⁡0.5/ln⁡(1−p)≈0.69/p\ln 0.5 / \ln(1-p) \approx 0.69/p. The approximation can be checked at several settings.

The calculation. Each page independently fails to inject with probability 1−p1-p, so all nn fail with probability (1−p)n(1-p)^n, and at least one succeeds with probability 1−(1−p)n1-(1-p)^n. Since ln⁡(1−p)≈−p\ln(1-p) \approx -p for small pp, this is about 1−e−np1 - e^{-np}: the risk depends on the product npnp, so doubling the workload has the same effect as doubling the per-page success rate.

A coding agent is a new identity in the environment and should be secured accordingly: least privilege, isolation, short-lived secrets, human approval for irreversible actions, and logging. Before an agent is given access, one question is useful: if a third party could write one paragraph that this agent will read, what is the worst action it could cause? The answer should then be reduced to an acceptable level, since over enough pages such a paragraph will eventually be written.

References

  1. J. H. Saltzer and M. D. Schroeder, "The protection of information in computer systems", Proceedings of the IEEE, 63(9), 1278-1308, 1975.
  2. OWASP, Top 10 for Large Language Model Applications, 2025 edition.
  3. NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023.
  4. B. D. Jovanovic and P. S. Levy, "A look at the rule of three", The American Statistician, 51(2), 137-139, 1997.
  5. ISO/IEC 42001:2023, Information technology - Artificial intelligence - Management system.