Jorge Quitério Search
IT & Engineering

A signed and scanned container pipeline for Docker hosts

Supply-chain security is usually explained with Kubernetes admission controllers, but many teams run Docker hosts. This article presents a reference design with GitLab CI, Harbor and Portainer that rejects unsigned or vulnerable images, and a simulation showing that a scan gate is a classifier with measurable error rates.

Most articles on software supply-chain security assume Kubernetes. The cluster refuses to run an image unless an admission controller has checked its signature and scan results. This assumes that a cluster is available.

Many teams do not have one. They run containers on a handful of Docker hosts managed with something like Portainer, and Docker runs any image it is instructed to run, including one built on a developer's laptop. There is no admission controller, so the checks have to live somewhere else.

This article gives a reference design for that setup, with GitLab CI, Harbor and Portainer, and then examines a component that most designs treat as straightforward: the "fail the build on critical vulnerabilities" gate. That gate is a statistical classifier, and like every classifier it makes errors at rates that can be calculated.

Several standards and papers address this problem.

NIST SP 800-218, the Secure Software Development Framework (2022), lists practices for producing secure software, including protecting the integrity of builds and releases and identifying vulnerabilities in components.

SLSA (Supply-chain Levels for Software Artifacts), an OpenSSF framework, defines increasing levels of build integrity and provenance.

Newman, Meyers and Torres-Arias (2022) describe Sigstore, which made code signing practical by replacing long-lived keys with short-lived certificates tied to an identity and recorded in a public transparency log. Cosign is its container-signing tool.

The EU Cyber Resilience Act, Regulation (EU) 2024/2847, requires manufacturers of products with digital elements to identify their components, including through a software bill of materials (SBOM), and to handle vulnerabilities over the product's lifetime. Its reporting obligations apply from 11 September 2026 and most other obligations from 11 December 2027; the text on EUR-Lex should be consulted for the applicable provisions.

On prioritisation, Jacobs and colleagues (2021) introduced the Exploit Prediction Scoring System (EPSS), now maintained by FIRST, which estimates the probability that a vulnerability will see exploitation activity in the next 30 days. CISA's Known Exploited Vulnerabilities (KEV) catalogue lists vulnerabilities already exploited.

The principle

Every image that reaches a host must be identified by its digest, scanned, described by an SBOM, and signed, and the deployment step must verify all of that before it changes anything.

Pharmacies in the European Union work on the same principle. Under the Falsified Medicines Directive, a prescription medicine is dispensed only if its pack carries an intact anti-tampering seal and a unique code that is checked against a central register; a pack that fails either check is not dispensed, however it reached the shelf. In the pipeline, the digest plays the role of the code, the signature that of the seal, and the deployment job that of the pharmacist.

The pipeline, stage by stage

  1. Static checks. Linting, static analysis (SAST) and secret detection.
  2. Dependency scanning. Software composition analysis on the lock files.
  3. Build. Once pushed, the registry digest (sha256:…) becomes the image's identity; tags are only labels.
  4. Image scan, for example trivy image --exit-code 1 --severity CRITICAL,HIGH.
  5. SBOM, for example syft <image> -o cyclonedx-json.
  6. Push to Harbor and record the digest the registry returns. That digest is what gets signed.
  7. Sign and attest: sign the digest with cosign and attach the SBOM as a signed attestation, not a loose file.
  8. Deploy by digest, after verification.

GitLab offers templates for several of these scans; some depend on the licence tier, which should be checked before relying on them.

Signing without managing keys

Cosign supports keyless signing: the CI job proves its identity with a short-lived OpenID Connect token. In GitLab that means requesting an ID token with the audience sigstore. Two caveats: keyless signing against the public Sigstore service works with GitLab.com, while a self-managed GitLab needs its own Sigstore instance or key-based signing; and the public transparency log records the signing identity, such as the project path.

sign:
  stage: sign
  image: alpine:3          # pin by digest in real use
  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore
  before_script:
    - apk add --no-cache cosign
  script:
    - cosign sign --yes "$IMAGE@$DIGEST"
    - cosign attest --yes --type cyclonedx --predicate sbom.json "$IMAGE@$DIGEST"

If keyless signing is not allowed, keep the key in a KMS or Vault, never on the runner's disk. In either case the digest should be signed, never a tag, because tags can be moved.

The role of Harbor

  • Scan on push with its built-in Trivy scanner.
  • Block pulls of images above a severity threshold.
  • Require signatures for a project.
  • Immutable tags, so v1.4.2 cannot be silently replaced.
  • Robot accounts per project with minimum permissions.

The deployment job as admission control

Before it touches the stack, the deploy job verifies the signature and the attestation against the identity of the pipeline allowed to produce them:

ID='^https://gitlab.com/example-group/app//'
ISSUER='https://gitlab.com'
cosign verify --certificate-identity-regexp "$ID" --certificate-oidc-issuer "$ISSUER" "$IMAGE@$DIGEST"
cosign verify-attestation --type cyclonedx \
  --certificate-identity-regexp "$ID" --certificate-oidc-issuer "$ISSUER" "$IMAGE@$DIGEST"

Only then does it write the pinned reference image@sha256:… into the stack and trigger Portainer, through a stack webhook or a Git-backed stack. Hosts never pull by tag. Small platforms are also exposed to supply-chain attacks, and a pipeline of this kind produces the evidence regulators require as part of normal delivery.

The scan gate as a classifier

"Fail the build on any critical vulnerability" sorts findings into block and allow. Its errors have names: a false positive blocks a build for a vulnerability that will never be exploited; a false negative lets through one that will.

CVSS measures severity, not likelihood. And only a small minority of published vulnerabilities are ever exploited. When the condition of interest is rare, even a reasonably accurate test produces mostly false alarms: this is the base-rate effect, the same arithmetic that affects screening tests for rare diseases. A gate that adds a likelihood signal such as EPSS or KEV membership separates the two groups better, and only a better-separating score improves both error rates at once.

The simulation below treats the scan gate as a classifier. When exploitation is rare, a gate based on severity alone blocks mostly harmless builds, and moving the threshold only trades one error for the other. A score with more signal improves both.

This is a stylised model, not real CVE data. Scores for findings that will and will not be exploited are drawn as two bell curves; their separation, d′d', measures how informative the score is.

Suggested settings:

  1. The severity-only gate. With the defaults (3% exploited, d′=0.8d' = 0.8, threshold 2), the gate catches only about 12% of exploitable findings and only about 14% of its blocks are real.
  2. Lower threshold. Move the threshold down to 1. Recall rises (to about 42%), but precision falls to about 8% and needless blocks increase, from about 22 to about 154 per thousand findings. Raising it does the reverse. Moving the threshold shifts the balance between the two errors without improving both.
  3. Stronger signal. Press "Severity + exploitation likelihood". At the same threshold, recall rises to about 58% and precision to about 44%.
  4. Base rate. Set the share of exploited findings to 0.5%, then to 20%. Precision falls sharply and then rises strongly, although the gate itself is unchanged. The base rate determines the result as much as the gate does.

The formulas. With non-exploited scores N(0,1)N(0,1) and exploited scores N(d′,1)N(d',1), a threshold tt gives recall 1−Φ(t−d′)1 - \Phi(t - d') and false-positive rate 1−Φ(t)1 - \Phi(t). With base rate π\pi, Bayes' theorem gives

precision=π [1−Φ(t−d′)]π [1−Φ(t−d′)]+(1−π) [1−Φ(t)]. \text{precision} = \frac{\pi\,[1-\Phi(t-d')]}{\pi\,[1-\Phi(t-d')] + (1-\pi)\,[1-\Phi(t)]} .

At π=0.03\pi = 0.03, d′=0.8d' = 0.8, t=2t = 2: recall =1−Φ(1.2)≈0.115= 1 - \Phi(1.2) \approx 0.115, false-positive rate ≈0.023\approx 0.023, so precision ≈0.0035/(0.0035+0.022)≈0.14\approx 0.0035 / (0.0035 + 0.022) \approx 0.14. Only a larger d′d' raises recall without raising the false-positive rate.

Without Kubernetes, the deployment job serves as the admission controller. It should verify a signature on a digest, a signed SBOM and scan results before Portainer changes anything. The scan gate should be treated as a classifier with error rates:

  • Combine severity with likelihood (EPSS, KEV) rather than repeatedly adjusting a severity threshold.
  • Measure the gate's false positives and false negatives from the pipeline's own history, with the same sampling arithmetic as in statistical sampling in control testing.
  • Keep exceptions explicit, time-limited and signed off by the risk owner.

References

  1. NIST SP 800-218, Secure Software Development Framework (SSDF) Version 1.1, 2022.
  2. OpenSSF, SLSA: Supply-chain Levels for Software Artifacts, specification v1.0, 2023: slsa.dev.
  3. Z. Newman, J. S. Meyers and S. Torres-Arias, "Sigstore: Software signing for everybody", Proceedings of the ACM Conference on Computer and Communications Security (CCS), 2022.
  4. Sigstore project, cosign documentation: docs.sigstore.dev.
  5. Harbor documentation: goharbor.io/docs.
  6. Regulation (EU) 2024/2847, the Cyber Resilience Act, Official Journal of the European Union.
  7. J. Jacobs, S. Romanosky, B. Edwards, I. Adjerid and M. Roytman, "Exploit Prediction Scoring System (EPSS)", Digital Threats: Research and Practice, 2(3), 2021.
  8. CISA, Known Exploited Vulnerabilities Catalog: cisa.gov.