Your Admission Webhook Never Saw That Container: Runtime Supply-Chain Verification With containerd's NRI

Your admission webhook never saw that container. If the sentence sounds wrong, check the bypass list: static pods managed directly by the kubelet, direct kubelet API access, a webhook outage that forces a choice between cluster lockup and silent bypass, a misconfigured namespace selector that quietly skips verification. Every one of these is a documented path past Kyverno, OPA Gatekeeper, or Sigstore Policy Controller — the tools most teams trust to verify image signatures and attestations at the Kubernetes API layer.

On July 30, 2026, Sascha Grunert published a CNCF blog post proposing the fix: move verification one layer down, into the container runtime itself, where every container must pass through regardless of how it was scheduled. The reference implementation is the Supply Chain NRI Plugin, a Node Resource Interface plugin that hooks container creation on CRI-O and containerd and verifies SLSA provenance, VEX, and VSA attestations before the container starts. This post walks through why the API layer leaks, how runtime verification works, what it takes to wire it across a Cluster-API-managed fleet, and the operational costs that decide whether it complements or replaces your webhook.

The gap below the API server​

Admission webhooks verify what the API server sees. The problem is what the API server never sees:

For a multi-tenant self-hosted PaaS, this table reads as a tenant-isolation problem. A platform that promises "every image on this fleet is verified" is making a claim its enforcement layer cannot keep: there is no label to forget at the runtime layer, no webhook network path to disrupt, and no gap between namespaces. Every container on the node passes through the runtime. That is the point you cannot skip — so that is where the second layer of enforcement belongs.

How runtime verification works​

The Node Resource Interface (NRI) is a plugin API supported by CRI-O 1.28+ and containerd 1.7+ (enabled by default since containerd 2.0). NRI plugins are long-lived daemons that talk to the runtime over a Unix domain socket and subscribe to container lifecycle events. When a plugin subscribes to CreateContainer, the runtime calls it synchronously before the container starts — and if the plugin returns an error, the container is rejected.

The Supply Chain NRI Plugin uses exactly this hook point. On every CreateContainer call it extracts the image reference and digest from runtime annotations, fetches supply-chain attestations from the OCI registry, verifies them against a per-namespace policy, and allows or rejects the container. Notably, verification happens at container creation, not at image pull — even an image pulled hours ago without an attestation check gets verified before it runs, including containers from pre-pulled or cached images.

Three attestation types are verified, each answering a different question:

Discovery runs through a single OCI Referrers API call per image digest (with a cosign tag-based fallback), and signatures are verified cryptographically with sigstore-go, supporting both keyless (Fulcio/OIDC) and key-based verification. This complements CRI-O's existing containers-policy.json, which validates image signatures but says nothing about attestation content like provenance, vulnerability status, or verification summaries.

One design note worth knowing: this started life as a proposal for an in-tree CRI-O feature, and community feedback pushed it toward a plugin instead — it works with both CRI-O and containerd, ships on its own release cycle, and keeps the runtime's critical path simple.

Walkthrough: wiring it across a Cluster-API fleet​

Rolling the plugin out on one node is a demo; rolling it out across a CAPI-managed fleet is the actual job. The plugin separates configuration into two layers by design: operational settings (a TOML file controlling mode, timeouts, cache, circuit breakers) and security policy (per-namespace JSON documents). An operator tuning cache TTLs should not have to dig through trust-root definitions, and the two layers change on different cadences anyway.

A minimal operational config looks like this:

verification = "enforce"
fetch_timeout = "30s"
fetch_failure_policy = "warn"
cache_ttl = "24h"
cache_failure_ttl = "5m"
policy_dir = "/etc/nri-supply-chain/policies"

And a per-namespace policy pins who you trust and what you require:

{
  "trust": {
    "issuers": ["https://token.actions.githubusercontent.com"],
    "sanPatterns": ["https://github.com/example-org/**"],
    "sources": ["github.com/example-org/*"]
  },
  "slsa": { "missingPolicy": "deny" },
  "vex": { "missingPolicy": "deny" }
}

Policies live one-per-namespace in policy_dir: default.json applies everywhere unless overridden, production.json applies to the production namespace, and namespace policies can inherit from the default or replace it entirely. For a multi-tenant PaaS, that per-namespace scoping maps cleanly onto tenants — strict provenance requirements for production tenants, permissive observation for dev sandboxes, infrastructure images excluded via glob patterns.

Before touching any node, dry-run a single image against the policy with no NRI connection required:

nri-supply-chain --config config.toml \
  --verify-image ghcr.io/example-org/app:1.4.2

The output reports per-check results (slsa, vex, or a short-circuiting vsa) plus an allowed verdict. A --validate flag checks config and policy files for errors without contacting any registry — the kind of check that belongs in CI before the policy ever reaches a node.

Deployment modes on CAPI-managed nodes​

The plugin ships as a single static binary, a multi-arch container image, and DEB/RPM packages, with three deployment paths that map differently onto a Cluster-API fleet:

Plugin updates follow the same path as the mode you chose: DaemonSet updates roll like any workload, while baked-in binaries roll with the node image. All configuration changes hot-reload via SIGHUP and filesystem watching with no pod restarts, and the verification cache is only cleared when policy-affecting fields actually change.

The phased rollout​

Supply-chain verification is not something you flip on overnight, and the plugin ships with verification = "disabled" by default. The intended path has three phases.

Phase one is observation: permissive policies (missingPolicy: "allow") with verification = "warn", so every container is checked but nothing is blocked. The nri_supply_chain_verification_total{result="fail"} Prometheus metric counts how many containers would be blocked — answering the first real question: what is our supply-chain posture right now?

Phase two tightens per namespace: production gets missingPolicy: "deny" for SLSA while dev stays permissive. Phase three switches to enforce, at which point the plugin logs warnings at startup if anything permissive remains (fetch_failure_policy still on warn, any missingPolicy still on allow). Failures surface in pod events as structured messages: supply chain verification failed: SLSA provenance: builder "X" not in trusted builders list.

What it costs: failure modes and trade-offs​

Every verification system has to decide what happens when things go wrong. The plugin makes these trade-offs explicit and configurable — here is the honest accounting.

Registry unreachable: fail open or fail closed? The fetch_failure_policy setting controls the behavior, and the default (warn) is explicitly fail-open: containers pass, a warning is logged, and the fetch-error counter increments. Setting it to deny trades availability for security — a registry outage then prevents new containers from starting. Per-registry circuit breakers amplify the trade-off: after consecutive failures to a host, fetches short-circuit for a configurable cooldown. In warn mode that means never-verified images from that registry bypass checks during the cooldown (previously verified images are still served from cache); in deny mode the breaker prevents cascading timeouts from blocking the whole node.

Container-start latency. Verification sits in the synchronous container-creation path, so latency engineering is load-bearing, not ornamental. Results are cached per image digest and namespace with configurable TTLs; on startup the plugin receives all running containers via NRI's Synchronize callback and pre-warms the cache in the background, so restarts don't cause a thundering herd of registry requests. Concurrent requests for the same image are deduplicated to a single verification, a semaphore caps parallel fetches, and retries use exponential backoff. The VSA short-circuit is the scaling story: a CI pipeline verifies once at build time, attaches a signed VSA, and every node validates that single attestation instead of re-running SLSA and VEX checks per container.

The NRI timeout gap. This one is subtle and the threat model is upfront about it: without extra runtime configuration, NRI fails open when the plugin is unregistered or when it answers slower than the runtime's NRI request timeout. The plugin's answer is an admission_timeout (default 1500ms) that rejects the container in enforce mode before the runtime's own timeout can admit it — plus required_plugins in the runtime's NRI config, which rejects containers while the plugin isn't registered at all. If you deploy this and skip required_plugins, you have built a gate with the hinges on the outside.

Tag-to-digest TOCTOU. When containerd doesn't provide a digest in annotations, the plugin resolves it via a registry HEAD request — and a malicious registry could serve a different digest than what the runtime pulls. The threat model rates this high severity with a blunt workaround: pin images by digest or use CRI-O, which isn't affected. For a PaaS, "pin by digest" is enforceable at the platform layer and probably already on your list.

Key and trust distribution to nodes. Every node needs the trust roots to verify anything: the TUF root pinning Fulcio CA certificates and Rekor public keys (a stale cached root remains usable for up to 24 hours if Sigstore is unreachable), plus registry credentials via the standard Docker credential chain (~/.docker/config.json and credential helpers) on each node. Policy files are protected by filesystem permissions, with an OCI-based policy source available to avoid local files entirely. Keyless (Fulcio/OIDC) verification sidesteps key-file management on nodes altogether, which is the path of least operational resistance.

The ceiling: node integrity. Runtime verification closes the gap between the API server and the container, but it assumes the node itself is intact. An attacker with root can kill the plugin process, replace policy files on disk, or disable NRI in the runtime config entirely. That is out of scope by design — and it is also the argument for pairing this with hardened, minimal node images rather than treating it as a substitute for them.

Complement or replace?​

After all of that, the question the title poses deserves a direct answer: this complements your admission webhook; it does not replace it. The CNCF post is explicit — webhooks remain the first layer, the NRI plugin the second — and the reasoning survives contact with the trade-off table:

• Keep the webhook for fast feedback at deploy time, rich policy expression (Kyverno/Gatekeeper do far more than image verification), and rejection before scheduling burns any node resources.
• Add the NRI plugin for the containers the webhook can never see: static pods, direct node access, outage windows, and selector misconfigurations. It is the enforcement point that cannot be skipped from the API layer.
• For a multi-tenant PaaS specifically, the per-namespace policy model is the feature that earns the second layer: tenant-scoped trust with default.json as the floor, VSA short-circuiting to keep per-node verification cheap at fleet scale, and a warn-first rollout that measures posture before it blocks tenants.

The honest version of "defense in depth" is two layers with different failure modes, not two copies of the same one. An API-layer check and a runtime-layer check fail differently — different blast radius, different bypass paths, different outage behavior — and that asymmetry is the whole point.

The bottom line​

Admission webhooks verify what the API server sees; an NRI plugin verifies what the runtime actually executes. If your threat model includes static pods, direct node access, or webhook outages — and on a multi-tenant platform, it should — runtime-level verification is the layer that closes those gaps. Start in warn mode, read the posture metric, tighten per namespace, and only then enforce. The gaps below the API server were always there; now there is a maintained, runtime-agnostic plugin sitting in the one path every container must take.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.